Transcription
TDP ArchitecturalSketches18-juni-2017
Subjects IntroductionGeneral information on the architectural sketchesLogical TDP ArchitectureDatapipes the elementary building blocksLink to generic big data patternsImplementation of 18-juni-2017TDP ExpressTDP AdvancedReisman use caseTDP Architectural Sketches
Introduction
Introduction Bert Dingemans Independent Data Architect and EA Consultant Solution architect for the TenneT Data Platform Author/Developer of Book Data Architectuur in de praktijk raktijk/9200000027961426)Web Publication Platform for Sparx EA (http://wpp.interactory.nl)Participant in http://eaxpertise.nl Working with EA since version 8 for 18-juni-2017Datamodeling (classes)XML message modelingArchiMate 2 & 3 for enterprise architectures and solutionarchitecturesArchitectural document generationTDP Architectural Sketches
Introduction TenneTTenneT is a leading European electricity transmission system operator(TSO), with activities in the Netherlands and in Germany. We strive toensure a reliable and uninterrupted supply of electricity (99,9999) inour high-voltage grid for some 41 million people. In doing so, we makeevery effort to meet our stakeholders' needs by being responsible,engaged and connected.18-juni-2017TDP Architectural Sketches
Facts & figures 3040 employees22573 km Grid15 HVDC stations523 Mln Euro profit1848 Mln Euro investments18-juni-2017TDP Architectural Sketches
Our grid18-juni-2017TDP Architectural Sketches
DTC and TDPData Transformation CorporateTenneT Data Platform18-juni-2017
DTC programmeExplanationDevelop the data space Smart meter data Production and storagedata (incl LV / MV) Grid planning data (incl MV) Congestion managementdata (incl MV) Improve TenneT’s visibilityas a data company thatcreates value for society, asa neutral party To support the first work stream,quickly and visibly take responsibilityfor enabling the change by publishingrelevant data in a user friendly manner As a second step, create an intelligentenergy market platform: a host for thenew energy eco systemBuild our internal data andanalytics capabilities Data and IT governance Analytics Infrastructure 18-juni-2017 Formally establish vertical leadershippositionInformally, as a front runner, establishhorizontal leadershipTeam tasksa.b.TDP Architectural Sketchesc.Run a set of internal businessd.improvement projects with a heavydata component, supported by externalspecialists, with the aim to enhancee.our internal capabilities on datagovernance, data management,f.analytics, infrastructureDefine a newmarket designand develop a directbusiness modelLobby for data accessand new marketdesignBuild thefront end of theintelligent data hubManage aportfolio ofprojectsScout for bestpractices and partnersBuild TenneT DataPlatform
Traditional data and big dataTraditional (deployed)OLTP dataDataWareHouseBig data (discovery)Hadoop &StreamingdataSocial dataTransactional dataInternal app dataLog dataSensor nUnstructuredExploratoryIterativeContext dataSituational dataERP dataMachine dataSCADA18-juni-2017TraditionalSourcesTDP Architectural SketchesNewSourcesPublic data10
Scope of TDPTraditional (deployed)Big data (discovery)Scope of TDPOLTP dataDataWareHouseHadoop &StreamingdataSocial dataTransactional dataInternal app dataLog dataSensor nUnstructuredExploratoryIterativeContext dataSituational dataERP dataMachine dataSCADA18-juni-2017TraditionalSourcesTDP Architectural Sketches26.09.201FusszeileNewSourcesPublic data11
Scope of TDPTraditional (deployed)Big data (discovery)Scope of TDPOLTP ivedataanalyticsTransactional dataInternal app dataLog dataSocial dataSensor nUnstructuredExploratoryIterativeContext dataSituational dataERP dataMachine dataSCADA18-juni-2017TraditionalSourcesTDP Architectural SketchesNewSourcesPublic data
TDP development cf. DAMHOF18-juni-2017TDP Architectural Sketches
TDP development cf. DAMHOFDWHMDMETLHadoopExpresslab18-juni-2017TDP Architectural SketchesHadoopAdvancedlab
General information on the architectural approach For the TDP we have a use case driven approach We use an agile approach for developing the platform architectureand support the use cases Just in Time architecting is necessary for the support of use cases We create a layered architecture tu support just in time architecting We use building blocks for standardisation and modelling speed We use ArchiMate as architectural modelling language and UMLclass diagrams for information modelling We use Sparx Enterprise Architect for continuous architecting Architectural documents are generated from the EA repository onrequest18-juni-2017TDP Architectural Sketches
18-juni-2017
Subjects in EA Logical architecture Big Data Patterns ABB Hourglass modelsData pipes SBB ExpressAdvanced Use cases ReismanRES forecasting Non functionals 18-juni-2017DamaISO Software qualitiesAICTDP Architectural Sketches
application Logical integration platform architectureData usageLayered architecture OverviewData security18-juni-2017Data managementData consuming systemsData processing in consuming systemsData storage in consuming systemsAPI & Gateway for data integrationData integration systemsData processing in integrationData storage in integration systemsAPI & Gateway for data sourcesData source systemsData processing in source systemsData storage in source systemsData productionTDP Architectural Sketches
Example storageapplication Data storageLegendCurrently relevantShort term relevantData storageRelational databaseData warehouseDatacube18-juni-2017File systemDistributed file systemDatamartTDP Architectural SketchesGeographical datastorageKey value storeLow relevanceNoSQLDocument databaseNewSQLColumnarQueues and stacksGraph
Example interfacesapplication Data integration interfacesODBCDatabase linkData accessProgrammers APIOLEDB and JDBCetcData integrationViews andmaterialized viewsApplicationintegrationFile transferSOAP/XMLwebservicePackages andstored proceduresWebserviceUser interfaceRelational interfaceGEO WebservicesB2B integrationReportDashboardForms and pagesGraphsUBL etcPortals and widgetsEDI18-juni-2017TDP Architectural SketchesODataebMS
Example management18-juni-2017application Data managementLegendCurrently relevantData ownershipData ManagementRegistryMeta datamanagementData managementRule mgtData modelingMDMTDP Architectural SketchesData qualitymanagementShort term relevantLow relevance
Link to Big DataPatterns18-juni-2017
Example data pipes18-juni-2017application Big Data PipelineLegendaCompoundBig Data Pipeline*RequiredOptionalPoly Source*Poly Storage*TDP Architectural SketchesBig Data ProcessingEnvironment*Poly Sink*Automated DatasetExecution
Example analyticalsandbox18-juni-2017application Analytical SandboxProcessing AbstractionLegendaDirect Data AccessAutomated DatasetExecutionCompoundRequiredOptionalLarge Scale BatchProcessingLarge Scale GraphProcessingAnalytical Sandbox*Poly Source*Confidential DataStoragePoly Storage*Data Size ReductionTDP Architectural Sketches
Data pipes18-juni-2017
Data pipe ABB overviewapplication Serv icemodel ABB18-juni-2017Data integrationUser interfaceApplicationintegrationRegister extractionData QualitiesGeneric Dataset AccessStandardized object ordatamodelConnection requirementsRelationaltransformationFile transformationImplementation processesMessagetransformationNoSql transformationStreamingtransformationGovernance processesUse cases en logischmodel uitwerkenwannneer welkeABB/SBB toepassenTDP Architectural Sketches
Data pipe ABB relationalapplication Relation transformation logical architecture ABB18-juni-2017Data targetModelingTransformationLogical modelingRelational transformationLoadPhysical modelingTransformWorkflow managementExtractData sourceRelational databasesTDP Architectural Sketches
Data pipe SBB messageapplication XML transformation logical and physical architecture (SBB)18-juni-2017Data targetSparx EnterpriseArchitect 12.1ModelingLogical modelingOracle DataIntegrator for BigDataTransformationOracle DataIntegratorEnterprise EditionRelational transformationManagement Packfor Data integrationLoadPhysical modelingTransformExtractWorkflow managementRelational databasesOracle Service BusXML TransformationIntermediate XMLstorageData sourceXML FileTDP Architectural SketchesOracle DataIntegrator forHadoopXML basedwebservice
TDP Express18-juni-2017
Logical Architectue Express18-juni-2017application TDP Express logical architectureData securityData usageReporting/BIData analyticsData managementVisualisationData consuming systemsData processing in consuming systemsData modelingAdministration,Authenticationand authorisationData TransformationEvent processingFiltering and selectionOrderingMeta datamanagementAnonimisationand maskingData storage in consuming systemsNoSQLDistributed file systemAPI & Gateway for data virtualisationFile transferTDP Architectural Sketches
Physical Architectue Expressapplication TDP Express logical and physical architecture18-juni-2017HUEApache OozieData usageReporting/BIVisualisationR & R StudioApache PigData securityData analyticsData managementApache SparkData consuming systemsData processing in consuming systemsApache SentryAdministration,Authenticationand authorisationData TransformationEvent processingFiltering and selectionOrderingData modelingAnonimisationand maskingData storage in consuming systemsNoSQLDistributed file systemHDFSAPI & Gateway for data virtualisationFile transferTDP Architectural SketchesSparx EnterpriseArchitect 12.1Meta datamanagementApache Impala
Express cloud alternativesapplication TDP Express cloud alternativ esapplication Hadoop on TPCapplication Hadoop in Public CloudApache HadoopApache HadoopBig Data PlatformServiceBig Data PlatformServiceHadoop modulesData StorageBig DatainfrastructurePublic cloudData ProcessingData StorageBig DatainfrastructureData ProcessingStorage DeviceHadoop NodeconfigurationProcessing DeviceVirtualized TPC infrastructureStorage Device18-juni-2017Hadoop NodeconfigurationProcessing DeviceTDP Architectural Sketches
TDP Advanced18-juni-2017
Logical Architectue Advanced18-juni-2017application TDP Adv anced logical architectureData securityData protection(privacy)PrescriptiveanalyticsPredictive analyticsData analyticsData miningdiscoveryData usageDescriptiveanalyticsPortalVisualisationData managementReporting/BIData qualitymanagementData consuming systemsData modelingData processing in consuming systemsAdministration,Authenticationand authorisationData TransformationEvent processingFiltering and selectionOrderingData tracing andlineageMDMData storage in consuming systemsGeographical datastorageRelational databaseDistributed file systemMeta datamanagementNoSQLData ownershipAnonimisationand maskingAPI & Gateway for data virtualisationFile transferTDP Architectural SketchesRelationalinterfaceWebserviceGEO WebservicesProgrammers API
Physical Architectue Advancedapplication TDP Adv anced logical and physical architecture18-juni-2017R & R StudioData securityData dictive analyticsData analyticsData miningdiscoveryData usageDescriptiveanalyticsPortalApache SparkVisualisationData managementReporting/BIData qualitymanagementApache OozieApache PigData consuming systemsData modelingData processing in consuming systemsAdministration,Authenticationand authorisationApache SentryHUESAP BOData TransformationEvent processingFiltering and selectionSparx EnterpriseArchitect 12.1Apache ImpalaOrderingData tracing andlineageMDMData storage in consuming systemsGeographical datastorageRelational databaseDistributed file systemMeta datamanagementNoSQLData ownershipAnonimisationand maskingAPI & Gateway for data virtualisationFile transferOracle GoldenGateOracle DataIntegratorEnterprise EditionTDP Architectural SketchesRelationalinterfaceHDFSWebserviceGEO WebservicesOracle Service BusProgrammers APIStreaming analytics
Reisman18-juni-2017
application Logical Architecture ReismanLogical Architecture ReismanData security18-juni-2017Data usageVisualisationData analyticsData managementData integrationReporting/BIData consuming systemsData processing in consuming systemsData storage in consuming systemsAPI & Gateway for data virtualisationData integration systemsData processing in integrationRelationaltransformationData storage in integration systemsData warehouseFile transformationAPI & Gateway for data integrationRelationalinterfaceFile transferData source systemsData processing in source systemsData productionSO & POVNB AbrufTDP Architectural SketchesVNBVNB MessWeather ll/Ecount dataScadaNLS08Prophet n
Lessons learned Agile approach is difficult when other teams stay waterfall LDAIMCBOnly ArchiMate for modeling is insufficientUsage of the quick scan in powerpoint helped a lotArchitectural maturity levels are different in GE and NLStart with a data catalog from the start (in EA?)Data Management and Governance are essential for successCooperate with other large programmes (Libra/BMR) within TenneT18-juni-2017TDP Architectural Sketches
More information http://wpp.interactory.nl is configured with the Big Pattern catalog Eap file is available from the zip Email for more information: bert@interactory.nl18-juni-2017TDP Architectural Sketches
DisclaimerAansprakelijkheid en auteursrecht TenneTDeze powerpoint wordt u aangeboden door TenneT TSO B.V. (“TenneT”). De inhoud ervan - alle teksten,beelden en geluiden - is beschermd op grond van de auteurswet. Van de inhoud van deze powerpointmag niets worden gekopieerd, tenzij daartoe expliciet door TenneT mogelijkheden worden geboden enaan de inhoud mag niets worden veranderd. TenneT zet zich in voor een juiste en actueleinformatieverstrekking, maar geeft ter zake geen garanties voor juistheid, nauwkeurigheid envolledigheid.TenneT aanvaardt geen aansprakelijkheid voor (vermeende) schade, voortvloeiend uit deze powerpoint,noch voor de gevolgen van activiteiten die worden ondernomen op basis van gegevens en informatie opdeze powerpoint.
www.tennet.euTenneT is een toonaangevende Europese netbeheerder (Transmission System Operator,TSO) met haar belangrijkste activiteiten in Nederland en Duitsland. Met circa 22.000kilometer aan hoogspanningsverbindingen zorgen we voor een betrouwbare en zekereelektriciteitsvoorziening aan 41 miljoen eindgebruikers in de markten die we bedienen.Taking power further
Hadoop & Streaming data New Sources Enterprise Integration Structured Repeatable Linear Unstructured Exploratory Iterative Log data Social data Situational data Context data Sensor data Public data Transactional data ERP data Internal app data OLTP data Traditional (deployed) Big data (discovery) SCADA Machine data Descriptive analytics .
