WIXLIB

Lessons Learned withGeodatabase ReplicationSCAUG ConferenceAustin, TexasApril 1, 2009ESRI – SCAUG 20091

Presentation Outline What is replication? City of New Braunfels Use Case Replication Demo Lessons Learned Q&AESRI – SCAUG 20092

What is Geodatabase Replication? Allows you to distribute copies of data across 2 ormore geodatabases You can edit the databases independently andsynchronize them as needed Released at 9.2 - Builds upon disconnected editingfrom earlier releases (8.3) NotN t ththe same as RDBMS replicationli tiESRI – SCAUG 20094

Geodatabase Replication Workflows Workflows can involve MultiMulti--User geodatabases and single usergeodatabases– MultiMMultilti-user geodatabased t b– MultiMMultilti-user ArcSDEA SDE geodatabased t baccesseddlocally or remotely through ArcGIS server– Single user geodatabase – Personal ArcSDE, file geodatabase orpersonal geodatabase on a local machineMultiMulti‐l ‐user to Singlel UserEnterprisePersonalGeodatabase,File Geodatabase orPersonal ArcSDEMulti‐‐user to Multi‐MultiMulti‐userEnterpriseEnterpriseESRI – SCAUG 20095

Geodatabase Replication Concepts Basic replication relationship is between two replicas :– A Child Replica is created from a Parent Replica.Parent Replica AChild Replica A You can replicate :– A specific version– SpecificSifi datasetsd tt– A subset of features in the chosen datasetsESRI – SCAUG 20096

Geodatabase Replication TypesCh k t wayParentChildOnce OnlyChildMultiple TimesChildMultiple TimesESRI – SCAUG 20097

Check--out / CheckCheckCheck--in Replication Called “Disconnected Editing” in ArcGIS 8.3 to 9.1 Child replica can be hosted in a Personal Geodatabase,Fil GeodatabaseFileG d t bor ArcSDEA SDE GeodatabaseG d t b(only( lArcSDE can host the Parent)Personal GDBEnterpriseGeodatabasePersonal SDEFile GDBESRI – SCAUG 20098

One--way ReplicationOne Child replica is considered readread-only– Child data may be overwritten No system versions on the child replica Choose between 2 model types:– Full – Supports complex types (Geometric Networks andTopologies)Tl i ) andd requiresitheth childhild replicaslidatad t tot beb versionedid– Simple – Child replica’s data is simple and does not need to beversionedWeb usersEnterpriseiGeodatabaseArcIMSArcGIS ServerRead‐onlyyGeodatabaseESRI – SCAUG 20099

Two--way ReplicationTwo Requires ArcSDE geodatabases and versioned data Can use 2-way replication with personal rsonal SDEPersonal SDEESRI – SCAUG 200910

Geodatabase Replication – Data Requirements You must have write access to the data All data must be registered as versioned without the option tomove edits to base TwoTwo--way and OneOne--way replicas– Each dataset must have a GlobalID column– All spatialpdata must be stored in a highg precisionpspatialpreference Use wizards in ArcMap or ArcToolbox to create or register replicasESRI – SCAUG 200911

Replica Synchronization Applying data changes across replicas Replicas can be Synchronized in either both directions or just asingle direction Synchronizationyis based on exchangingg g messagesg and is faulttolerant Uses versioningTransferESRI – SCAUG 200912

Replica Synchronization Distributed Geodatabase Tools– ArcMap toolbar or ArcToolbox Connected Synchronization– Message exchange is automatic Disconnected Synchronizationy– Message exchange is performed by the end user– Operations are performed by export, file transfer and import Example: to synchronize, export changes to a delta XML file, transfer the file(ftp, CD through the mail, etc.), have the file imported on the relative replicawhen it arrivesESRI – SCAUG 200913

Working through errors System is designed to stay consistent If the system fails during a synchronization, it is rolled back to it’sprevious state If a data changes message is lost in a disconnected system, thenext messageg will contain changesg from the lost messageg and anyynew changes Replica log can be used to get error information about asynchronizationESRI – SCAUG 200914

Requirements ArcSDE 9.2 or above on both sides– Child replicas must be same or newer version than parent– Personal SDE is sufficient (SQL Server Express)– OneOne--way replication to PGDB and FGDB supported at 9.3 ArcEditor or ArcInfo on both sides ArcGIS Server Standard or Advanced Edition (if creating GeodataWebeb Service)Se ce) Collaboration and Agreement– Common data model at creation time PatienceESRI – SCAUG 200915

Business Case Enterprise License Agreement– ArcInfo desktopp clients– ArcSDE & ArcGIS Server Advanced to publish Web ADF site Serving 35,000 map requests monthly Utilize 911 District for street centerline data Relational database structure for street centerlines and addresspoints (StreetNameID key and tblStreetName lookup table) Use spatial views to manage display of attributes Use a locator service built upon streets for geocoding Limited staff resources OtherOth managementt systemstcomingionlineli (l(land/asset,d/t pavement)t) A lot rides on our streets!ESRI – SCAUG 200917

Previous Update Process Bexar Metro 9-1-1 Network District– Merged separate PGDBs– Loaded to FTP site City of New Braunfels– Download PGDB from FTP site– Stop all published map services– Kick all users out of ArcSDE– Create temporary backup of feature class and street name table– Visually confirm new data structure matched existing– Overwrite entire feature class and table (convert feature class fromPGDB to ArcSDE, table converted through XML workspace)– Reapplypp y securityyppermissions– Delete and recreate spatial views– Delete and recreate locator services– Restart and verify all services operational– Archive old feature class and table– Delete temporary backups from ArcSDEESRI – SCAUG 200918

OLD PROCESSESRI – SCAUG 200919

Desired Update Process The City of New Braunfels desired a process for theupdate of street centerline and street name changeswhich:– provided a direct import of data (no more conversion)– allowed for uninterrupted map and database services– took full advantage of the City’s existing GIS investments– reduced network traffic– maintained a consistent data structure– could be automatically scheduled– reduce data management workload The answer: geodatabase replicationESRI – SCAUG 200920

New ProcessESRI – SCAUG 200921

Production Environment New Braunfels– ArcGIS Server 9.3 SP1 Geodata service– Firewall– ArcSDE 9.3 SP1 on MS SQL Server 2005 Bexar Metro 911 Network District– Firewall– ArcSDE 99.33 SP1 on MS SQL Server 2000ESRI – SCAUG 200922

Initialization Bexar Metro 911 Network District– Migrate from PGDB to ArcSDE– Registergas versioned– Generate GlobalIDs– Export ArcSDE Feature Dataset usingXML Workspace City of New Braunfels– ImportIt XML WorkspaceW ktotArcSDE– Register as versioned– Set security privileges– Create a QA/QC version– Create and publish geodataservice– Create replica to XMLmessage– Import replica XML message– Edit replication connectorESRI – SCAUG 200923

Result The City of New Braunfels desired a process for theupdate of street centerline and street name changeswhich:provided a direct import of data (no more conversion)allowed for uninterrupted map and database servicestook full advantage of the City’s existing GIS investmentsreduced network trafficmaintained a consistent data structurecould be automatically scheduledreduce data management workload FormerlyFl a monthly,thl manuall updated t process whichhi htook 22-3 hours and interruption of all servicesypprocess takingg 15 minutes,, even Now,, a weeklymanually, and no interruption of servicesESRI – SCAUG 200924

ReplicationDemoTransferESRI – SCAUG 200926

Geodatabase Replication – Getting Started Anticipate future needs when defining the data toreplicate Have a well defined data model before creating replicas Choose the right replica type– Consider 22--way replicas with personal ArcSDE instead ofcheck--out replicascheck– Use 11--way replicas over 22-way replicas when possibleESRI – SCAUG 200928

Geodatabase Replication – Getting Started Use models or scripts for replicas you plan to createand synchronize on a regular basis– You can use the Create Replica and Create Replica From Servergeoprocessing tools in ModelBuilder Consider using the following replica creation options– ReRe--use schema (check(check--out replicas) – uses existing schema– Register only – replicates prepre-copied data– Relationship classes processing is optional Schedule Synchronizations– You can use geoprocessing models exported to Python or VBscript and the Windows Scheduler– Consider synchronization orderESRI – SCAUG 200929

Lessons Learned Requires thorough prepre-planning–Common data structure–Not all feature classes or attribute fields need bereplicated–PrePre--seed data on both databases–Ensure IT cooperation on both sides! TEST! TEST! TEST!ESRI – SCAUG 200931

Lessons Learned Creator of the replica is always the parent of the replica Data provider may or may not be the parent of thereplicali Geodata service must be published using local dataowner account– Enable ArcGIS Server security or use firewall Only the replica parent can replicate to version otherth DEFAULTthan OneOne--way replication must be from parent to childESRI – SCAUG 200932

Lessons Learned Not quite “setset it and forget itit”– Vast improvement over Sneakernet Data updated without interruption of services SmallerS ll message traffict ffi– Have a plan for recreating replica with reseeding data– Overcome with manual transmission of XML messages– Use generation levels to help troubleshoot– Still, reduced maintenance Requires spirit of cooperation and trust betweenentitiesESRI – SCAUG 200933

Resources Introduction to Geodatabase Replication(free online .cfm?DID 6&Product ID 867 2008 ESRI User Conference DVDs ArcGIS Server 9.x Web Help How to find help on Geodabase Replicationhttp://support.esri.com/index.cfm?fa knowledgebase.techarticles.articleShow&d 34131 Using Compress on ArcSDE GDBs with atabase/Blog/GDB Rep Compress.pdf TCP/IP Tuning:http://support.esri.com/index.cfm?fa knowledgebase.techarticles.articleShow&d 35971ESRI – SCAUG 200934

Questions?Thank you!!ESRI – SCAUG 200935

RDBMS vs. Geodatabase ReplicationGeodatabase ReplicationPurpose: Data Distribution One- or Two-wayArcSDDE technoloogy All or some feature classesArcObjects Portions of feature classes : HA (HighAvailability) Data DistributionAvailability), One-way only Full GeodatabaseOS/StorageESRI – SCAUG 200938

DBMS Replication with Geodatabases Geodatabase replication does not use DBMSreplication RequirementsRit andd limitationsli it tioff DBMS replicationli ti– Requires knowledge of how the geodatabase\geodatabase\ArcSDE systemtables work– No tools provided in ArcGIS to support it– Limited support for cross DBMS replication– Doesoes notot supportsuppo t oro hasas limitedted supportsuppo t foro complexco p egeodatabase data types and limited filters to define the data toreplicate Advantages of DBMS replication– Can work with nonnon-versioned data– Can replicate entire database– Can be configured to provide synchronous replicationESRI – SCAUG 200939

Geodatabase Replication - Data Requirements You must have write access to the data All data must be registered as versioned without theoptionti tot move editsdit tot baseb TwoTwo--way and OneOne-way replicas– Each dataset must have a GlobalID column– All spatial data must be stored in a high precision spatialreferenceESRI – SCAUG 200940