Transcription
ELK Stack:Elasticsearch, Logstash and KibanaKajorn Pathomkeerati (IAI)Samuel Ambroj Peréz (SCC)INSTITUTE FOR APPLIED COMPUTER SCIENCE (IAI), FACULTY OF INFORMATICSSTEINBUCH COMPUTING CENTER (SCC)KIT – University of the State of Baden-Wuerttemberg andNational Research Center of the Helmholtz Associationwww.kit.edu
Extra: Sense (Beta)Extension for Google Chrome, available in Chrome Web StoreUser-friendly consoleDesigned for ElasticsearchIndividual RequestText Highlight,Auto CompleteIndentationRequest2Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaResponseInstitute for Applied Computer Science, Faculty of Informatics
Visualization Tool For ElasticsearchKIBANA3Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Kibana - OverviewFull integration with ElasticsearchEasy ConfigurationImport & DiscoveryTime-based DataReal-time DiscoveryVisualizationEasy to customizeFast analyticDashboardConnecting the visualizationsEasy to share as iframe or URLOpen source. Community driven. Apache licensed.More Infohttps://www.elastic.co/products/kibana4Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Kibana- Live DemoKIBANA GOES LIVE5Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Kibana - SummaryAdvantagesEasy visualizingVarious visualizations availableFully integrated with ElasticsearchLimitationsNo custom aggregation supportedNo custom requestEvent-based data onlyElasticsearch data onlyDashboard built on saved visualizationsDashboard filter affects all visualizations6Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Use CaseELASTICSEARCH& SQL DATABASE7Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
OverviewRelational Database:Traditional SQL DatabasesComplex SQL-Statements needed for some analyticsStill widely usedElasticsearch:Non-relational Databases - NoSQLAs-a-serviceAccessible via HTTPRelational DBà Non-relational DBData MigrationUsing plugin : JDBC River plugin8Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
JDBC RiverAn Elasticsearch PluginEnabling data migrationSQL Database à ElasticsearchImport – functionUsing SQL statement to filter dataUsing a JDBC connectorSupports native connectorsMySQL, Postgresql, .Note:River plugin is deprecated since ES 1.5 (Currently 1.7)Still supported by community9Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
JDBC River - Parameters curl –XPUT 'localhost:9200/ river/type name/ meta' -d'{"type" : "jdbc","jdbc" : {"url" : "jdbc:mysql://localhost","user" : "db user","password" : "db user password","sql" : "SELECT * FROM table name","index": "es index","type" : "es type","type mapping": { }}}'Easy importFilter Data by SQL-StatementMore 10Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Experimental ScenarioInput:A dump file for MySQLOutput:Visualizations in KibanaQuestion: How?Answer: following instructions1. Prepare MySQL Server2. Prepare JDBC River plugin for MySQL3. Import data to Elasticsearch4. Visualizing with Kibana11Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
MySQL Server - InstallationRequired Components / File:MySQL ServerMySQL ClientAn SQL dump FileInstructions:1. Set up MySQL Server & Client aptitude install mysql-server mysql-client2. Create a database mysql -u root –p create database db name; exit3. Restore the database with dump file mysql -u root –p db name dump file.sql12Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
JDBC River - InstallationRequired Components:JDBC River PluginJDBC Driver (Connector)Instructions:1. Download & install JDBC River plugin2. Download & install a JDBC driver3. Restart Elasticsearch13Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
JDBC River - Installation (2)Download & install JDBC River plugin cd /usr/share/elasticsearch ./bin/plugin --install river-jdbc -url rch-river-jdbc-1.5.0.5-plugin.zip'Download & install a JDBC connector(found in MySQL JDBC driver) cd /usr/share/elasticsearch/plugins/ /mysql-connectorjava-5.1.36.tar.gz tar -zxvf mysql-connector-java-5.1.36.tar.gz --wildcards '*.jar‘ mv .1.36-bin.jar./river-jdbc/ rm -rf mysql-*Restart Elasticsearch Service /etc/init.d/elasticsearch stop /etc/init.d/elasticsearch start14Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Import DataDefine JDBC River parameters curl -XPUT 'localhost:9200/ river/tweet/ meta' -d'{"type" : "jdbc","jdbc" : {"url" : "jdbc:mysql://localhost:3306/tweetsql","user" : "root","password" : "root","sql" : "select tid as id, tweet, hashtag.hashtag, lang, created atfrom tweetleft join hashtag tweet on tweet.id hashtag tweet.tweet idleft join hashtag on hashtag.id hashtag tweet.hashtag id","index": "tweetsql","type" : "tweet","type mapping": {"tweet": {"dynamic": true,"properties": {"created at":{"type": "date","format": "EEE MMM dd HH:mm:ss Z yyyy"}}}}}}'15Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Visualization in Kibana16Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
PluginsOther River Plugins:Google Drive River PluginDropbox River PluginWikipedia River Plugin Cloud Service Discovery PluginsAWS Cloud Plugin, GCE Cloud Plugin, Analysis PluginsICU Analysis Plugin, Stempel Analysis Plugin, There are more rch/reference/current/modules-plugins.html17Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Exercise - MySQL DatabaseGiven:A dump file contains tweet dataGoal:Visualizations in KibanaExample for visualizationsNumber of tweets in totalNumber of tweets by a languageTop hashtags / tweet-languagesetc.Create a dashboard with various visualizations18Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Question ?19Kajorn Pathomkeerati - ELK Stack: Elasticsearch Logstash KibanaInstitute for Applied Computer Science, Faculty of Informatics
Elasticsearch, Logstash and Kibana Kajorn Pathomkeerati (IAI) Samuel Ambroj Peréz (SCC) 2 Institute for Applied Computer Science, Faculty of Informatics Extra: Sense (Beta) Extension for Google Chrome
