Transcription
Polyglot persistence: utilizingopen source databases as aSwiss pocket knifeArt van Scheppingen - Senior (No)SQL DBA @ VidaXLBart Oleś - Senior Support Engineer @ Severalnines
Who are we?
VidaXL Company Stats Online retailer in (mostly) slow moving goods Founded 2008 350M turnover, 40% growth yearly 1500 employees (US, CN, AU, IN, RO, UA) HQ in the Netherlands 4 warehouses worldwide (NL, US and AU)
How does VidaXL sell its goods? Own webshop platform in EU, US and AU Warehouses in NL, US and AU Selling on other platforms, e.g. Amazon, eBay Allow selling on our own platform using Mirakl B2B drop-shipments
VidaXL Technical Foundations SAP as ERP system Genesys as CS system Webshop Open source web-based development strategy PHP / NodeJS Docker Cloudflare workers
VidaXL DevOps Datastores MySQL MariaDB (Galera) clusters MySQL replication PostgreSQL SOLR Elasticsearch ELK MongoDBCouchbase(RabbitMQ)Prometheus
What is Polyglot Persistence?Using multiple specialized persistent stores rather than one single general-purpose database
Where does the term come from? The way we work is changing Enterprise applications are becoming more complex Separate (devops/agile) teams Ownership of applications (Micro)servicesEveryone has their preference Various programming languages Various storage systems
Where does the term come from? Monoglot Programming Only one programming language allowed Readability Support One platform to supportKnowledge All code is in the same languageEverybody is an expertIs there a jack-of-all-trades language?
Where does the term come from? Monoglot Programming Only one programming language allowed Readability Support One platform to supportKnowledge All code is in the same languageEverybody is an expertIs there a jack-of-all-trades language?
Monoglotprogramming
Monoglot programmingCarpenters actually use a broad variety of tools
Polyglot Programming Polyglot Programming Use programming languages for what they are good atFlexibility Use Java for a secure APIUse Scala for real time stream processingUse Python for text analysisTie everything together using AngularJSKnowledge Everybody is expert at one or more languages
Polyglot Programming
Monoglot Persistence
Data storage landscape changes Relational data stores (RDBMS)Key-Value data stores (“NoSQL”)Columnar data stores (OLAP)Document data stores (NoSQL)Graph data stores (GDB)Big Data
Data storage landscape changesSoftwareRDBMSOracle, MySQL,PostgreSQLKey-ValueRedis, RiakColumnarInfiniDB, ClickhouseDocumentMongoDB,CouchbaseGraphNeo4J, JanusgraphBig DataHadoop
Data storage landscape changesSoftwareAWSGoogleRDBMSOracle, MySQL,PostgreSQLRDS, AuroraCloudSQL, SpannerKey-ValueRedis, RiakDynamoDBDatastoreColumnarInfiniDB, eSimpleDBBigtableGraphNeo4J, JanusgraphNeptuneBig DataHadoopEMRCloud Dataproc
Even Hadoop has become a polyglot
Polyglot Persistence Complex problems require different storage systemsUse the right tool for the job, for example Use PostgreSQL for financial dataUse MySQL for website contentsUse MongoDB for user profilesUse Cassandra for real time streamsUse Neo4J for recommendation analysis
Use the right tool for the right jobDocument storage: MongoDB
Use the right tool for the right jobColumnar storage: Cassandra
Use the right tool for the right jobGraph storage: Neo4J
Polyglot Persistence
Polyglot Persistence at VidaXLYes we certainly are polyglots!
Quick recap on our data stores MySQL MariaDB (Galera) clusters MySQL replication ProxySQL PostgreSQLSOLRElasticsearch ELK MongoDBCouchbase(RabbitMQ)Prometheus
How did this happen? Continuous growth Hardly any time to overhaul existing systems Transition from monolith to microservice architecture For each microservice the most optimal solution has been chosen Early adopters of new technology Gaining advantage over competition
From monolith to microservice
From monolith to microservice
From monolith to microservice
From monolith to microservice
From monolith to microservice
From monolith to microservice
What were the challenges? Automation Increased complexity Systems monitoring Multiple integrations Maintenance becomes more difficult Backups Scaling Software updates DevOps are not a DBA
What were the solutions? Invest in automation Never perform any (large) task thrice Increase tooling Build it ourselves costs time Buying/licensing tools costs money Keeping the headcount low saves money Focus on systems that matter most Get (exteneral) help Hire DBAs! ;)
ClusterControl by Severalnines some subtitle here?
Thank you
Polyglot Persistence Complex problems require different storage systems Use the right tool for the job, for example Use PostgreSQL for financial data Use MySQL for website contents Use MongoDB for user profiles Use Cassandra for real time streams Use Neo4J for recommendation analysis
