Mastering Data Access With The Optic API & Template -Driven .

1y ago

22 Views

1 Downloads

1.10 MB

27 Pages

Report/dmca

Download PDF

Transcription

Mastering Data Access with theOptic API & Template-Driven ExtractionErik Hennum, Principal Engineer, MarkLogicFayez Saliba, Staff Engineer, MarkLogic13 June 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. COPYRIGHT

Introductions§§SLIDE: 3Erik Hennum-MarkLogic Principal Engineer (6 years)-Experience: Client APIs – Java, REST, Node.jsFayez Saliba-MarkLogic Staff Engineer (2 years)-Experience: Template-driven extraction COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Why Is This Hard Today?Kayla the Customer§ Who is Kayla? What is our relationship with her?§ Is ”Kayla” on the phone the same “Kayla” thatordered 3 products from us?§ What has she purchased? Returned? Receivedsupport on?§ Where does she live? How do I contact her?§ Is she a “high-value” customer? What risk doesshe represent?SLIDE: 4 COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Address: LondonKayla the CustomerCRM 1 – Where does Kayla live?OrdersEmail:kayla@spam.comSLIDE: 5Support Call 143CRM 2 Marketing – How do wecontact Kayla?ERP – Is this the same “Kayla” thatordered 3 products from us?Support – What has Kaylareceived support on? COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Building on the MarkLogicFoundationSLIDE: 6§Load from any source in its natural form§No pre-defined schema required§Entities, messages, hierarchical data naturallyas documents§Integrated search enables discovery-Expose data using lenses-Perform flexible data query COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

The 360 Call CenterCustomerOrderProductIssue§Data from disparate relational data sources§Data cleansing in MarkLogic after ingestion-OrderLine§IssueHistorySLIDE: 7Easy to model with Entity ServicesData set for board games – but could be-Health care – Patient, Procedure, -Financial services – Trade, Counterparty, COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Customer 360 Challenge§Support Team needs to understand the Customer 360 – to take the appropriate next step:-Route the incoming customer call to the appropriate Support Personnel – based on orderhistory and customer status-Provide supplier-specific detail about products to improve engagement-Review the support ticket transcript to evaluate customer service interactionKAYLASLIDE: 8 COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

MARKLOGIC 9Easier Data Query With Templates and the Optic APIBENEFITSOPTIC APISEARCHSQLSPARQLLENSES(TEMPLATES)DOCUMENTS(JSON OR XML)SLIDE: 9§ Single, integrated platform– single, unified queryinterface§ Access any shape of data,or use all together§ Combine strengths of eachunderlying querymechanism COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Optic API§§Language-integrated, multi-model joinsand aggregates over rows and triplesFluent JavaScript, XQuery, and Java-leverage your language skills-avoid string concatenation or injection-modularize with variables or functionsop.fromView( ).where( ).select( ).joinInner(op.fromView( ), op.on( , )).groupBy( , op.count( , )).where( ).orderBy( )§Extract from or construct documents.limit( )§Same engine, indexes as SQL/SPARQL.result();-SLIDE: 10same d-node pushdown optimizations COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Translate Challenges Into Optic API QueriesCustomer 360 ChallengeOptic API CapabilityWhich customers ordered the most Joins & aggregatesproduct (by count and price)?Which supplier-specific productdetail matches these terms?Document queryReview the support ticket transcript Document joinsto confirm that top customers aretreated wellSLIDE: 11 COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

From Source Documents To Indexed RowsThe row structures projected from a document are stored in the indexDocument: /products/1000055.json{"SKU": "101591922267","title": "different gallon",“price": 4.5,. }View: productsSKUtitleprice101591922267 different gallon 4.5 § Project the names and data types of the columns§ Project the values of columns in the rowsSLIDE: 12 COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Join and Aggregate on the RowsDeclarative joins and aggregates – find the top customers by order volumecustomersordersidlast namesales region1BURDETTE Nevada2LYONMarylandorder date customer titleprice3/21/2017gothic cuckoo15.00weird air23.7913/21/2017joined customersand orders92 id last name sales region92 1THOMASNevadaBURDETTENevada order datecustomer3/21/201713/21/20171top customers1 BURDETTE last nameNevada3/21/2017 sum price1order count 430.81SAYERS11KING SLIDE: 138464.78 title price gothicreliablecuckoonephew15.0014.36reliable nephew14.36 COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Query on the Documents Providing the RowsQuery on the source document for rows– find products based on a variant detail price101591922267 different gallon131841112651gothic cuckoo159929577929 reliable nephew SLIDE: 14 4.515.0014.36 {"SKU":"101591922267","title":"different gallon",“price":4.5,.“detail":{"features": "Humorous","endorsements": "Games "}} COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Joins on Document URIsJoin documents based on uri – retrieve chat log for issue to assess support for top customerIssue history/issue-chats/40.xml support-chat chat-transcript conceptual cobweb /issue-chats/40.xml staff not functionalcustomer and most recent timestamp 2017-03-27T14:36:28 /timestamp issue with transcript message Hello, how can I help you? /message first name last name titletranscript EmmaSCHMIDT conceptual cobweb support-chat not functional chat-transcript staff timestamp 2017-03-27T14:36:28 /timestamp message Hello, how can I help you? /message titleSLIDE: 15chat COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Where Does the Data Come From?§SLIDE: 16Introducing Template-Driven Extraction (TDE)Customer 360 ChallengeTDE CapabilityThe Customer Director has differentneeds from the customer support agent.Queries have different pieces of data.Support for multiple viewsAnswering detailed questions requires agranular view of the data – but eachorder contains many lines, each with itsown product and price.Handling repeating rowsTwo entity types (apparel & eyewear)describe product accessories, but needefficient aggregation across both.Limited Transformationduring Indexing COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Simple Workflow§Where in the lifecycle does Template generation fit?Template-Driven ExtractionDATA LoadedData loaded into MarkLogicHarmonize/Enrich if desiredEntity ModelAuto-Generated TemplatesTemplate Driven Extraction Fine tune a template Build custom templatesSLIDE: 17§ Cover advanced way ofbuilding a custom templatefrom scratch§ Show how templates canindex views and triples fromdocuments§ Demo some powerfulfeatures like transformationin templates COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Documents for OrdersTemplate-DrivenExtractionIndexed “Orders” -22"order date":"2017-04-22",2017-04-30"ship date":"2017-04-30""lines":[{"product id":"1000213","price":35,"quantity":2,"discounted price":35,"title":"meaningfulwedding"}],}SLIDE: 18{"template":{/id"context": teger","val": "./customer"},.customer order date ship date166 962 2017-04-22 2017-04-30 345 214 2017-03-12 2017-03-17 236 78 2017-02-04 2017-02-10 SELECT *FROM customer360.ordersWHERE op.fromView("customer360","orders").result() COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Repeating Order Lines{"lines":[product id 1000213{ "product ounted price":33.5,golden clam"},clam"title":"golden{ "product id":"1000109",product id ted price":6.29,White : product id"context":"/lines/product ines","columns":[{"name":"order e":"product id","scalarType":"integer","val":"./product id"},.]}Indexed VieworderidProductidpricequantitydiscounted price166 1000213 35 2 33.5 166 1000109 8.99 3 6.29 124 1000234 2.5 18 2.2 124 1000589 45 2 42.5 345 1000623 32 4 31 SELECT FROM ordersWHERE JOINorderlines ON GROUP BY COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Template-Driven ExtractionFlexible Data Projection From Documents Into IndexesProject data from documents into the indexCreate SQL Views or semantics TriplesQuery from SQL, SPARQL, ODBC, or OpticSimple language with context-based projectionData Projection§§§§Transactional &Data Provenance§ Rows and Triples are updated with a document update§ Document is intact, not transformed nor modifiedMultiple Projectionsand Transformation§ Transforming data on indexSLIDE: 20§ One document many ViewsMany data sources one view COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Different DocumentsBoard Game{"game id": "1000085","SKU": "101897294018","title": "wonderfulfootnote","price": 25,"years active": ":"/game id",.}Single ProductsViewid1000085 42 Game Accessory{"id": "42","sku": "416739731622","title": "Cards","price": 97294018 wonderful footnote 416739731622 Cards2510 }SLIDE: 21 COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

DocumentsTemplate-DrivenExtractionTriple IndexGame Accessory{"id": "42","sku": "416739731622","title": "Cards","price": 10,"game ID": "1000146"}SLIDE: 22 template . context /id / . triple subject val sem:iri( prefix-product ./id) / predicate val sem:iri( prefix-pred "isAccessoryFor") / object val sem:iri( prefix-product ./game ID) / /triple . / PREFIX prefix-pred: http SELECT ?product ?accessoryWHERE { ?accessory . }op.fromTriples([op.pattern(.),op.pattern(.)]) COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

DEMO

Template-Driven Extraction Highlights / Recap§Indexed views and triples:-Updated with documents. ACID-Inherent Backup, Replication, Failover, and so on-Document level Security inheritedIngest or Re-index§No physical extraction, copying, changes to the underlying documentNO ETL§One source with multiple viewsTRANSACTIONAL§Multiple sources with one viewMULTIPLE VIEWS§Update your template with Schema changes§Single document management§SLIDE: 24Templates actively keep the projections up-to-dateMULTIPLE SOURCESREPEATING ROWSCONTEXT AWARE COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Optic API Highlights / Recap§The MarkLogic-idiomatic, language-integrated interface to the SQL / SPARQL engine-§Relational operations-§Row joins and grouping with aggregatesDocument-oriented operations-SLIDE: 25Access to rows, triples, and range indexes as well as documentsConstraining queries, document joins, XPath extraction, node and sequence construction COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Integrated Queries Over Your Data§Transformative integration of multiple data sources and structures by query-§What are the barriers for your organization?§SLIDE: 26In the call center scenario, breaking the barrier to a new level of customer engagementHow could integrated query transform your enterprise? COPYRIGHT 2017 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Q&A

Optic API & Template -Driven Extraction Erik Hennum, Principal Engineer, MarkLogic Fayez Saliba, Staff Engineer, MarkLogic . REST, Node.js § Fayez Saliba- . Easier Data Query With Templates and the Optic API MARKLOGIC 9 §Single, integrated platform -single, unified query interface