Transcription
TECHNICAL DOCUMENTData quality monitoring andsurveillance system evaluationA handbook of methods and applicationswww.ecdc.europa.eu
ECDC TECHNICAL DOCUMENTData quality monitoring andsurveillance system evaluationA handbook of methods and applications
This publication of the European Centre for Disease Prevention and Control (ECDC) was coordinated byIsabelle Devaux (senior expert, Epidemiological Methods, ECDC).Contributing authorsJohn Brazil (Health Protection Surveillance Centre, Ireland; Section 2.4), Bruno Ciancio (ECDC; Chapter 1, Section2.1), Isabelle Devaux (ECDC; Chapter 1, Sections 3.1 and 3.2), James Freed (Public Health England, UnitedKingdom; Sections 2.1 and 3.2), Magid Herida (Institut for Public Health Surveillance, France; Section 3.8 ), JanaKerlik (Public Health Authority of the Slovak Republic; Section 2.1), Scott McNabb (Emory University, United Statesof America; Sections 2.1 and 3.8), Kassiani Mellou (Hellenic Centre for Disease Control and Prevention, Greece;Sections 2.2, 2.3, 3.3, 3.4 and 3.5), Gerardo Priotto (World Health Organization; Section 3.6), Simone van der Plas(National Institute of Public Health and the Environment, the Netherlands; Chapter 4), Bart van der Zanden (PublicHealth Agency of Sweden; Chapter 4), Edward Valesco (Robert Koch Institute, Germany; Sections 3.1 and 3.2).Project working group members: Maria Avdicova (Public Health Authority of the Slovak Republic), Sandro Bonfigli(National Institute of Health, Italy), Mike Catchpole (Public Health England, United Kingdom), Agnes Csohan(National Centre for Epidemiology, Hungary), Yves Dupont (Scientific Institute of Public Health, Belgium), IrenaKlavs (National Institute of Public Health, Slovenia), Anna Kurchatova (National Centre of Infectious and ParasiticDiseases, Bulgaria), Mathias Leroy (Scientific Institute of Public Health, Belgium), David Mercer (World HealthOrganization, Regional Office for Europe), Zsuzsanna Molnar (National Centre for Epidemiology, Hungary), PierreNabeth (World Health Organization), Elvira Rizzuto (National Institute of Health, Italy), Malgorzata Sadkowska(National Institute of Public Health, Poland), Gudrun Sigmundsdottir (Centre of Health Security and CommunicableDisease Prevention, Island), Ewa Staszewska (National Institute of Public Health, Poland), Reinhild Strauss (FederalMinistry of Health, Austria).Contributing ECDC experts: Cristian Avram, Arnold Bosman, Sergio Brusin, Bruno Ciancio, Denis Coulombier,Isabelle Devaux, Erika Duffell, Ana-Belen Escriva, Rodrigo Filipe, Graham Fraser, Keith Hodson, Frantiska Hruba,Charles Johnston, Marius Mag, Vladimir Prikazsky, Carmen Varela-Santos.AcknowledgementsWe would also like to thank the following individuals for their help, input and feedback: Elliot Churchill, LarisaFedarushchanka, Lisa Ferland, Carmen Emily Hazim (all Public Health Practice, LLC; United States of America),Christopher Williams (Public Health England, United Kingdom), Michaela Dierke (Robert Koch Institute, Germany).Suggested citation: European Centre for Disease Prevention and Control. Data quality monitoring and surveillancesystem evaluation – A handbook of methods and applications. Stockholm: ECDC; 2014.Stockholm, September 2014ISBN 978-92-9193-592-5doi 10.2900/35329Catalogue number TQ-04-14-829-EN-N European Centre for Disease Prevention and Control, 2014Reproduction is authorised, provided the source is acknowledgedii
TECHNICAL DOCUMENTData quality monitoring and surveillance system evaluationContentsAbbreviations . vii1 Introduction .11.1 Background .11.1.1 Monitoring data quality .11.1.2 Evaluating surveillance systems .21.2 Surveillance attributes .21.2.1 Completeness and validity .21.2.2 Sensitivity, specificity, positive predictive value and negative predictive value .31.2.3 Timeliness .51.2.4 Usefulness .51.2.5 Representativeness .51.2.6 Other surveillance attributes .61.2.7 Development of data quality indicators .61.3 Description of a surveillance system .71.3.1 Surveillance objectives .81.3.2 List of diseases under surveillance and case definitions .91.3.3 Data sources and data flow . 101.3.4 Surveillance networks . 111.3.5 Populations under surveillance. 121.3.6 Geographic coverage . 121.3.7 Types of surveillance . 121.3.8 Specification of the information to be reported . 131.3.9 Reporting format . 141.3.10 Data entry . 141.3.11 System architecture . 151.4 References . 162 Monitoring data quality . 172.1 Conceptual framework . 172.1.1 Underreporting and under-ascertainment . 172.1.2 Setting up a data quality monitoring system . 192.2 Monitoring completeness . 232.2.1 Describe the system . 232.2.2 Plan the monitoring process . 252.2.3 Measure internal completeness and summarise results . 262.2.4 Interpret the results of the measurement . 272.2.5 Make suggestions for remediation . 282.3 Monitoring internal validity . 322.3.1 Describe the system . 322.3.2 Plan the monitoring process . 332.3.3 Measure internal validity and summarise the results . 342.3.4 Interpret the results of the measurement . 362.3.5 Make suggestions for remediation . 372.3.6 Questions and answers . 392.4 Monitoring timeliness . 402.4.1 Describe the system . 402.4.2 Plan the monitoring process . 412.4.3 Measure timeliness . 422.5 References . 443 Evaluating surveillance systems . 453.1 A stepwise approach to evaluation . 453.2 Plan the evaluation . 453.2.1 Identifying triggers . 453.2.2 Selecting the evaluation type and method . 483.2.3 Engaging stakeholders . 503.2.4 Setting up a team for surveillance evaluation . 503.3 Evaluation of external completeness . 513.3.1 Introduction. 51iii
Data quality monitoring and surveillance system evaluationTECHNICAL DOCUMENT3.3.2 Describing the system. 523.3.3 Planning the evaluation. 523.3.4 Evaluating external completeness . 543.3.5 Interpreting the results of the evaluation . 583.3.6 Improving external completeness . 593.4 Evaluation of sensitivity, specificity and positive predictive value . 603.4.1 Factors which influence the sensitivity of a surveillance system . 613.4.2 Methods . 613.4.3 Results . 633.4.4 Interpreting the results of an evaluation . 633.4.5 Enhancing sensitivity, specificity and positive predictive value . 643.4.6 Questions and answers . 643.5 Evaluation of external validity . 673.5.1 Identifying duplicate records . 673.5.2 Calculating concordance through record linkage . 673.5.3 Calculating concordance via follow-up of notified cases . 693.5.4 Validity checks during statistical analysis . 693.5.5 Validity of outbreak notifications . 693.5.6 Indirect methods to evaluate validity . 703.5.7 Ethical considerations . 703.5.8 Interpretation and modifications . 703.5.9 Questions and answers . 723.6 Evaluation of usefulness . 733.6.1 Introduction. 733.6.2 Identifying expected uses of a disease surveillance system . 733.6.2 Creating a user list . 733.6.3 Documenting system usage . 743.6.4 Remediation . 743.7 Evaluating the costs of disease surveillance . 753.7.1 Background . 753.7.2 Classification of costs of public health surveillance . 753.7.3 Cost analysis . 763.7.4 Discussion . 773.8 References . 794 Intervention . 844.1 Motivation of data providers for improving data quality . 844.1.1 Why improve data quality? . 844.1.2 How to motivate data providers to deliver good quality data . 844.2 Improving surveillance system quality . 854.2.1 How to motivate for surveillance system improvement . 854.2.2 Reinforcing monitoring. 864.2.3 Structuring output activities. 874.3 References . 89Glossary . 90iv
TECHNICAL DOCUMENTData quality monitoring and surveillance system evaluationFiguresFigure 1.2a. Levels of case detection in the surveillance process for communicable diseases.5Figure 1.3a. Key elements for the description of a surveillance system .8Figure 1.3b. Simplified flowchart of a generic surveillance system (indicator-based) . 11Figure 2.1a. Possible steps to identify non-visible cases who do not seek healthcare . 18Figure 2.1b. General practitioners, case management and surveillance systems . 20Figure 2.2a. Steps designing and implementing a monitoring system for internal completeness . 23Figure 2.2b. Checklist: main considerations for the establishment of a monitoring system . 25Figure 2.2c. Methods at the data entry level for enhancing internal data completeness . 30Figure 2.3a. Monitoring internal validity . 32Figure 2.3b. Checklist: main considerations/decisions about internal validity when setting up a monitoring system33Figure 2.3c. Surveillance systems-related factors which can negatively influence the validity of surveillance data. 37Figure 2.3d. Basic concepts for creating a reporting form . 38Figure 2.3e. Examples of validity checks . 38Figure 2.4a. Steps for monitoring timeliness . 40Figure 2.4b. Scope of a reporting system to measure timeliness . 41Figure 2.4c. Steps for monitoring timeliness . 42Figure 2.4d. Monitoring timeliness: main time delays . 42Figure 2.4e. Important timeliness elements in the Irish CIDR system . 43Figure 3.1a. Evaluating surveillance systems . 45Figure 3.2a. Selecting an appropriate evaluation type. 48Figure 3.3a. Key points for the evaluation of external completeness . 51Figure 3.3b. Venn diagram for a two-source capture–recapture study . 56Figure 3.3c. Venn diagram for a three-source capture–recapture study . 57Figure 3.4a. Key points for the evaluation of sensitivity, specificity and positive predictive value . 61Figure 3.5a. Key points for evaluating external validity . 67Figure 3.5b. Schematic presentation of the steps of a study of concordance between surveillance data and medicalrecords (external data source) . 68Figure 3.5c. Approaches to increase data validity in a disease surveillance system . 72Figure 4.1a. Key points . 84Figure 4.2a. Key points . 85Figure 4.2b. Process to reinforce monitoring of data quality of surveillance system . 87v
Data quality monitoring and surveillance system evaluationTECHNICAL DOCUMENTTablesTable 1.1a. Completeness of HIV data reported by countries of the WHO European Region (total: 53) in 2009 and2010 .2Table 1.1b. Completeness of data on Mycobacterium tuberculosis infection, EU Member States, 2010 .2Table 1.2a. Two dimensions of completeness and validity .3Table 1.2b. Sensitivity, specificity, positive predictive value, negative predictive value for a case reportingsystem/outbreak reporting system .3Table 1.2c. Examples of data quality indicators .6Table 1.3a. Data sources for indicator-based surveillance . 10Table 2.1b. Validity rules for data collected from the registered medical practitioner (RMPs) in the United Kingdom . 21Table 2.2a. Checklist of background information needed when designing a monitoring system for internalcompleteness . 24Table 2.2b. Table summarising the monitoring plan for internal completeness . 26Table 2.2c. Table summarising the monitoring plan for internal completeness of common variables . 26Table 2.2d. Examples of fields that are not completed at initial notification . 27Table 2.2e. Completeness results by variable. 27Table 2.2f. Completeness results by record . 27Table 2.2g. Checklist of possible reasons for low completeness of the variable 'culture result' for tuberculosisnotification . 28Table 2.2h. Examples of possible mandatory fields by disease. 30Table 2.3a. Checklist: necessary background information when designing a monitoring system for internal validity . 33Table 2.3b. Examples of validation rules and validation messages in TESSy . 34Table 2.3c. Number of notified cases and expected number of cases for a specific time period (e.g. week, month)and geographic area (e.g. whole country, region); based on background data . 35Table 2.3d. Proportion of notifications of specific diseases and expected proportion of notifications for a specifictime period and geographic area; based on background data. 35Table 2.3e. Reported incidence and expected disease incidence in a specific population for a given time period;based on background data. 35Table 2.3f. Number of recognised false alarms/errors by health outcome for a specific time period . 36Table 2.3g. Validity results by health outcome and variable for a specific time period . 36Table 2.3h. Validity results by health outcome for a specific time period . 36Table 3.2a. Examples of triggers leading to an evaluation . 46Table 3.2b. Triggers for the evaluation of surveillance systems: suggested definitions and categories. 47Table 3.2b. Types of protocols . 48Table 3.2c. Examples of attributes to be evaluated and possible type of evaluation for specific triggers . 49Table 3.3a. Important information to be included in the evaluation protocol . 52Table 3.3b. Important information to be included in the evaluation protocol . 54Table 3.3c. Checklist of possible reasons for low external completeness of surveillance systems . 58Table 3.4a. Sensitivity (Se) and positive predictive value (PPV) of surveillance system X. 63Table 3.5a. Evaluation of validity of data in surveillance system X, comparison between surveillance data and datafrom a reference source . 68Table 3.5b. Absolute mean difference between dates in the notification dataset and in the medical record (oralternative data source) . 69Table 3.5c. Validity of reports of outbreaks of specific diseases . 69Table 3.5d. Summary of the validation process . 71Table 3.6a. Sample user list for data generated by the health surveillance system . 73Table 3.8a. Conceptual framework of public health surveillance . 78vi
TECHNICAL DOCUMENTData quality monitoring and surveillance system evaluationAbbreviationsCDCUnited States Centers for Disease Control and PreventionLIMSLaboratory information management systemNOIDSNotifications of Infectious DiseasesNPVNegative predictive valueNSCNational surveillance coordinatorPPVPositive predictive valueSeSensitivitySpSpecificityTESSyThe European Surveillance Systemvii
TECHNICAL DOCUMENTData quality monitoring and surveillance system evaluation1 IntroductionThe European Surveillance System (TESSy) collects data on 52 infectious diseases and conditions collected throughnominated disease-specific networks from 30 countries, whose disease surveillance systems differ substantially.This leads to problems with regard to data quality and comparability which can influence the quality of analysis ofsurveillance data.Differences in notification rates are another factor that complicates data analysis. Differences in notification ratesbetween countries can be explained by epidemiological factors, but also by differences in the national surveillancesystems.Data comparability at the national, sub-national and EU level can be improved by collecting appropriate informationon the characteristics of surveillance systems, the quality of the data they generate, and their performance.1.1 BackgroundIn 2010, ECDC conducted a survey [1] among national surveillance coordinators (NSCs) which revealed substantialdifferences in practices with regard to monitoring data quality in EU Member States. The results
Data quality monitoring and surveillance system evaluation TECHNICAL DOCUMENT iv . Checklist: main considerations/decisions about internal validity when setting up a monitoring system 33 Figure 2.3c. Surveillance systems-related factors which can negatively influence the validity of surveillance data .37
