Transcription
Google Cloud Platform Solutionsfor DevOps EngineersMárton Kodok / @martonkodokGoogle Developer Expert at REEA.net - Targu MuresMarch 2019 - Vilnius, Lithuania
About me Geek. Hiker. Do-er.Among the Top3 romanians on Stackoverflow 130k reputationGoogle Developer Expert on Cloud technologiesCrafting Web/Mobile backends at REEA.netBigQuery/Redis and database engine expertActive in mentoring and IT communityStackOverflow: pentium10GitHub: pentium10Slideshare: martonkodokTwitter: @martonkodokGCP Solutions for DevOps Engineers@martonkodok
Agenda1.2.3.4.5.6.7.8.Application development in the CloudApp Engine: Scale your apps seamlessly from zero to planet scaleCloud Functions: your gateway to GCP ServicesGoogle Stackdriver: Metrics, logging, alerting are a universal right!BigQuery: federated data access warehouseDoing hybrid cloud mixing on premise with cloudPractical use casesQwiklabsGCP Solutions for DevOps Engineers@martonkodok
What’s so hard about traditional app erOrchestrationGCP Solutions for DevOps Engineers@martonkodok
REEA.net uses GCPBuild on the same infrastructurethat powers Google
Google Cloud Platform (GCP)ComputeBig DataIdentity & ueryCloudDataflowCloudDataprocCloudDataprepCloud IAMKeyCloud Resource Cloud ctionsContainerOptimized CorpData LossIdentity-Aware Security KeyPrevention APIProxyEnforcementInternet of ThingsCloud IoTCoreMachine LearningStorage & DatabasesCloud MachineLearningCloudVision APICloudSpeech APICloud DatastoreCloud NaturalLanguage APICloudTranslationAPICloudJobs APIAdvancedSolutions LabCloud SQLCloudSpannerPersistentDiskTransferAppliance
Google Cloud Platform (GCP)NetworkingManagement rtingTraceVirtualPrivate CloudCloud LoadBalancingCloudCDNCloudExternal soleCloudShellCloudInterconnectCloud DNSCloudNetworkCloudRoutesCloudFirewall RulesCloudRouterCloud VPNDedicatedInterconnectContainerBuilderDeveloper ToolsCloud MobileAppCloudBilling APIProfilerCloud SDKCloudTools forPowerShellCloudDeploymentManagerCloud SourceRepositoriesCloudTools forAndroid StudioCloud Toolsfor IntelliJCloudTools forVisual StudioContainerRegistryGoogle Plug-infor EclipseCloud TestLab
Google sees serverless asProgramming modelOperational modelFocus on codeZero opsEvent-drivenAutomatic scalingStatelessManaged securityDevGCP Solutions for DevOps EngineersOpsBilling modelPay for usage @martonkodok
Serverless is more than a set of functionsBigQueryCloud StorageCloud DataflowCloud TasksCloud FunctionsStackdriverGCP Solutions for DevOps EngineersApp EngineCloud PubSub@martonkodok
Meet Serverlessserverless data center depictedGCP Solutions for DevOps Engineers@martonkodok
Serverless is about maximizing elasticity, costsavings, and agility of cloud computing.GCP Solutions for DevOps Engineers@martonkodok
Serverless typesPlatformsGCP Solutions for DevOps EngineersTriggered Code@martonkodok
App Engine - managed application platformAPI 1MultipleClients App EngineAPI RequestsCloud LoadBalancingSplit App Versions API 2 Compute EngineVirtual MachinesScale your applications seamlessly fromzero to planet scaleAutomatically scales depending on yourapplication trafficTraffic Splitting (app versions, A/B tests,incremental rollouts)API 3Container EngineKubernetes ServicesGCP Solutions for DevOps EngineersBest used: For HTTP services For existing applications@martonkodok
App Engine: Services and rsionInstanceGCP Solutions for DevOps k
App Engine: Traffic splittingA/B testing andCanary releasingwith a few clicks ora single gcloudcommandGCP Solutions for DevOps Engineers@martonkodok
App Engine: RuntimesJava 8GCP Solutions for DevOps Engineers1.11@martonkodok
Async message processingGCP Solutions for DevOps Engineers@martonkodok
Asynchronous task executionCloud TasksMessage queue systemGuaranteed at-least-once deliveryFuture SchedulingWill only be dispatched once on dupsBest used: For HTTP services For App Engine handlersGCP Solutions for DevOps EngineersCloud PubSubDeliver event data based on topicsExactly-once processingBuild multi-cloud on premise, hybrid appsCross zone message replicationBest used: For large-scale ingestion of events, streams Topics, publish/subscribe patterns, IoT@martonkodok
Cloud Tasks - Message Queue systemsStatic ContentCloud StorageDynamic ContentFront End AppCloud LoadBalancingApp EngineDynamic ContentAutoscalingCloudDNSBatch AppCloud SQLApp EngineCloud ksCompute EngineCompute EngineAutoscalingGCP Solutions for DevOps Engineers@martonkodok
Task QueuesPush queues Put with delaysPull queues Ability to “tag” Lease multiple eg: gameboardupdated, game id as tag.Slides: https://www.slideshare.net/martonkodokTitle: Architectural Patterns - Message QueuesGCP Solutions for DevOps Engineers@martonkodok
Reduce request latencyGCP Solutions for DevOps Engineers@martonkodok
Cloud FunctionsGCP Solutions for DevOps Engineers@martonkodok
Cloud FunctionsTriggered CodeFrontendPlatform ServicesEvent ultMetrics / Logs/StreamingGCP Solutions for DevOps Engineers@martonkodok
Cloud FunctionsUnit: FunctionTrigger: Events and HTTPBest used: For Events & Async workloads For single-purpose microservicesGCP Solutions for DevOps Engineers@martonkodok
Cloud Functions - Event-driven - FaaS Simplest way to run your code in the cloud - just deployAutomatically scales, highly available and fault tolerantNo servers to provision, manage, patch or updatePay only while your code runsConnects and extends cloud services(In alpha: Java)Node 8.11Support for async/await Python 3.7.1Flask microframeworkGCP Solutions for DevOps Engineers Go 1.11Familiar building blocks@martonkodok
Functions: your gateway to GCP ServicesAccess 20 Google services from GCFServicesBigQueryApplicationsAPIsGCP Solutions for DevOps EngineersCloud MachineLearningCloud NaturalLanguage APICloudPub/SubCloudStorageCloudBigtableCloud VisionAPICloud SpeechAPICloudTranslation sistant@martonkodok
Cloud Functions TriggerCloud StorageCompute Engine EAPCloud Pub/SubFinalize/CreateInstanceTopicDeleteInstance GroupArchiveAutoscalerMetadata UpdateFirewallSet LabelsGCP Solutions for DevOps Engineers@martonkodok
Cloud Functions TriggerFirebaseBigQueryCloud FirestoreRealtime DB changeJob createCreateAuthenticationJob completeUpdateRemote configWriteGoogle Analytics logDeleteGCP Solutions for DevOps Engineers@martonkodok
Summary: When to use whatCloud FunctionsApp EngineServerless add-onSmallest unit ofcomputingUnit of computingis appsEvent drivenarchitectureHTTPrequest/responseRun functions, apps& containers onGKEConnect & extendservicesLarge scalablebackendsGCP Solutions for DevOps EngineersFull portability ofyour artifactsRun on your owncluster@martonkodok
The unit of computing is yours to chose!Slice and dice your application any way youthink it makes sense to be more agile, to buildbetter apps that deliver value.GCP Solutions for DevOps Engineers@martonkodok
How I Learned How to Stop Worryingand Dig Hosted MonitoringGCP Solutions for DevOps Engineers@martonkodok
What to do for monitoring? In-house or hosted?Modern vs proven?Specialized or general?Paid vs free?Let’s be honest: Not enough time Small environment More important work to do Everyone else is doing itGCP Solutions for DevOps Engineers@martonkodok
Google StackdriverIntelligent monitoring and managementfor services, containers, applications, andinfrastructure. GCP Solutions for DevOps EngineersWorks with GCP, AWS, on premLogging, tracing, alertsCollectd agentCost per resourceEasy point and click alerting policyDebugger, Error reporting, profiler *freeUptime checks *free@martonkodok
Google Stackdriver: Uptime check (offered free) Check Type: HTTP, TCPResource Type: URL, Instance, App Engine, Load BalancerGlobal LocationsCustom Headers (encrypted)AuthenticationWhitelisted source IPsAlerting policy: Email, SMS, Slack, PagerDuty, Hipchat, Campfire, WebhooksGCP Solutions for DevOps Engineers@martonkodok
Google Stackdriver: Uptime checkGCP Solutions for DevOps Engineers@martonkodok
Life of a logGCP Solutions for DevOps Engineers@martonkodok
Google StackdriverMetrics, logging, alerting are a universal right!GCP Solutions for DevOps Engineers@martonkodok
GCP Solutions for DevOps Engineers@martonkodok
What is BigQuery?Analytics-as-a-Service - Data Warehouse in the CloudScales into Petabytes on Managed Infrastructure - load up to 5TB large filesSQL 2011 Javascript UDF (User Defined Functions)Familiar DB Structure (table, columns, views, struct, nested, JSON)Integrates with Google Sheets Cloud Storage Pub/Sub connectorsDecent pricing (storage: 20/TB cold: 10/TB,queries 5/TB)GCP Solutions for DevOps Engineers*March 2019@martonkodok
Architecting for The CloudOn-Premises ServersFrontendPlatform ServicesPipelinesEvent SourcingETLEngineBigQueryMetrics / Logs/StreamingGCP Solutions for DevOps Engineers@martonkodok
BigQuery: federated data access warehouseApplication & PresentationAudit logsBilling cessingMLCloudStorageDatabaseSQLBI InterfaceData Studio 360FrontendPlatform ServicesReport & ShareBusiness AnalysisReal-Time EventsMultiple PlatformsGoogleMarketingPlatformGCP Solutions for DevOps Engineers@martonkodok
“Data needs to be processed inmultiple services.How can we pipe to multiple places?GCP Solutions for DevOps Engineers@martonkodok
Architecting for The CloudProcessOn-Premises ServersAnalyzeDataStudioFrontendPlatform ServicesThird-PartyToolsBigQueryStreamEvent SourcingCloudDataflowCloud SQLBatchMetrics / Logs/StreamingCloud StorageGCP Solutions for DevOps Engineers@martonkodok
“We have our app outside of GCP.How can we use the benefits of BigQuery?GCP Solutions for DevOps Engineers@martonkodok
Data Pipeline Integration at REEA.netDevelopmentOn-Premises ServersTeamFrontendPlatform ServicesLoad / ExportReplayarchiveEvent SourcingStandardDevicesHTTPSReport & ShareBusiness AnalysisCloud StorageMetrics / luentDQlikViewApplicationDatabaseServersServersGCP Solutions for DevOps EngineersSQLAnalytics BackendBigQueryData StudioInternalDashboard@martonkodok
The following slides will present a sample Fluentd configuration to:1.2.3.4.Transform a recordCopy event to multiple outputsStore event data in File (for backup/log purposes)Stream to BigQuery (for immediate analyses)GCP Solutions for DevOps Engineers@martonkodok
filter frontend.user.* @type record transformer /filter match frontend.user.* @type copy store @type forestsubtype file /store store @type bigquery /store 1Filter plugin mutates incoming data. Add/modify/deleteevent data transform attributes without a code deploy.2The copy output plugin copies events to multiple outputs.File(s), multiple databases, DB engines.Great to ship same event to multiple subsystems.34 /match GCP Solutions for DevOps EngineersThe Bigquery output plugin on the fly streams the event tothe BigQuery warehouse. No need to write integration.Data is available immediately for querying.Whenever needed other output plugins can be wired in:Kafka, Google Cloud Storage output plugin.@martonkodok
1record transformer2copy filter frontend.user.* @type record transformerenable rubyremove keys host record bq {"insert id":" {uid}","host":" {host}","created":" {time.to i}"}avg {record["total"] / record["count"]} /record /filter GCP Solutions for DevOps Engineers3file4BigQuerysyntax: Ruby, easy to use.Great for:- date transformation,- quick normalizations,- calculating something on the fly,and store in clear log/analytics db- renaming without code deploy.@martonkodok
1record transformer2copy3file4BigQuery match frontend.user.* @type copy store @type forestsubtype file template path /tank/storage/ {tag}.*.logtime slice format %Y%m%d /template /store /match GCP Solutions for DevOps Engineers@martonkodok
1record transformer2copy3file match frontend.user.* @type bigquerymethod insertauth method json keyjson key /etc/td-agent/keys/key-31da042be48c.jsontime field timestamptime slice format %Y%m%dtable user %{time slice}ignore unknown valuesschema path /etc/td-agent/schema/user login.json /match GCP Solutions for DevOps Engineers4BigQueryConnector uses:- JSON key auth file- JSON table schemaPro features:- streaming to Partitioned tables- ignore unknown values(not reflected in schema)@martonkodok
Our benefits using BigQuery SQL JSON (big costs saving with partitioning/clustering)run raw ad-hoc queries (either by analysts/sales or Devs)ability to throw in / join all kind of datapricing model 1TB free every monthno more throwing away-, expiring-, aggregating old datano provisioning/deployno running out of resourcesno more focus on large scale execution planGCP Solutions for DevOps Engineers@martonkodok
Qwiklabs - Find Your Topic and Skill LevelKubernetesMachine LearningCertificationIntroductoryEnroll in the Baseline: InfrastructureQuest.1. Kubernetes Engine: Qwik Start (lab)(video) - Deploy a containerizedapplication with Kubernetes Engine.2. IoT: Qwik Start (lab) - Learn aboutGCP’s Cloud IoT Core service, wherealmost any IoThing is possible!Enroll in the Baseline: Data, ML, AI Quest.1. Cloud ML Engine: Qwik Start (lab)(video) - Train and deploy a TensorFlowmodel to Cloud ML Engine.2. Cloud Natural Language API: QwikStart (lab) - Perform sentiment analysis ona block of text. For fun, paste in the lastemail you sent!Enroll in the Cloud Architecture Quest.1. Stackdriver Qwik Start: Web UI (lab)What’s up with your VM? Learn to installmonitoring and logging agents to collectinfo about your GCP resources.2. Getting Started with Cloud KMS (lab) Create your first KeyRing and CryptoKeywith GCP.ExperiencedEnroll in the Kubernetes Quest.1. Kubernetes Engine: Qwik Start (lab)(video) - Deploy a containerizedapplication with Kubernetes Engine.2. Intro to Docker (lab)Are your containers running? Create,run, and debug containers, then pull andpush images to and from GoogleContainer Registry.Enroll in the Machine Learning APIs Quest.1. Cloud ML Engine: Qwik Start (lab)(video) - Train and deploy a TensorFlowmodel to Cloud ML Engine.2. Detect Labels, Faces, and Landmarksin Images with the Cloud Vision API (lab)- How do you feel about self-driving cars?Learn what’s behind the AI making thefuture possible.Enroll in the Cloud Architecture Quest.1. Stackdriver Qwik Start: Web UI (lab) What’s up with your VM? Learn to installmonitoring and logging agents to collectinfo about your GCP resources.2. Application Performance Management(APM) with Stackdriver (lab) - Dig deeperinto Stackdriver with a problematicenvironment - can you identify the issues?For GCPnewcomers1 hour duration,2 labsQwiklabsalumni1 hour, 2 labsGCP Solutions for DevOps Engineers@martonkodok
Thank you.Slides available on: slideshare.net/martonkodokReea.net - Integrated web solutions driven by creativity to deliverprojects.
GCP Solutions for DevOps Engineers @martonkodok Cloud Tasks Cloud PubSub Message queue system Guaranteed at-least-once delivery Future Scheduling Will only be dispatched once on dups Best used: For HTTP services For App Engine handlers Deliver event data based on topics Exactly-
