Shared'Memory'Layer'for'SparkApplications - Linux Foundation Events

Transcription

gnite.incubator.apache.orgApache ,ApacheIgnite,Ignite akyan

Agenda AboutIn- ‐MemoryComputing crossSparkJobsIn- s Q&AApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

ApacheIgniteforBIandAnalyticsApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

ApacheIgniteIn- ‐MemoryDataFabric:StrategicApproachtoIMCTM Supports Applications ofvarious types andlanguages OpenSource – Apache 2.0 Simple Java APIs 1 JAR Dependency High Performance & Scale Automatic Fault Tolerance Management/Monitoring Runs on Commodity Hardware Supportsexisting &new data sources No need to rip & replaceApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

ApacheIgnite:BetterCloudSupport ithJClouds(NEW) DockerSupport– AutomaticallyBuildandDeployApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

DataGrid:JCache(JSR107) r)EventsandMetricsPluggablePersistence es(ANSI99)In- ‐MemoryIndexesAutomaticRDBMSIntegrationApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

DataGrid:PartitionedCacheApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

DataGrid:ReplicatedCacheApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

DevOps:IntegrationwithYarnandMesos ionEasyDataCenterConfigurationOn- ‐DemandElasticityApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

DataGrid:Off- ‐HeapMemory usesSmallOn- ‐HeapFootprintLargeOff- ‐HeapFootprintOff- nApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

ShareRDDsAcrossSparkJobs IgniteRDD– ShareRDDacrossjobsonthehost– ShareRDDacrossjobsintheapplication– ShareRDDglobally FasterSQL– In- ‐MemoryIndexes– SQLontopofSharedRDDApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

IgniteIn- ‐MemoryFileSystem IgniteIn- ‐MemoryFileSystem(IGFS)– Hadoop- ‐compliant– EasytoInstall– On- ‐HeapandOff- ‐Heap– CachingLayerforHDFS– Write- ‐throughandRead- ‐throughHDFS– PerformanceBoostApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

IgniteIn- ‐MemoryMapReduce In- ‐MemoryNativePerformance ZeroCodeChange UseexistingMRcode UseexistingHivequeries NoNameNode NoNetworkNoise In- ‐ProcessDataColocation EagerPushSchedulingApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

DataGrid:Ad- ‐HocSQL(ANSI99) ANSI- ‐99SQLAlwaysConsistentFaultTolerantIn- ‐MemoryIndexes(On- ‐HeapandOff- - ‐CacheJoins,Unions,etc.Ad- ‐HocSQLSupportApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

SQLCross- ‐CacheJOINExampleApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

SQLCross- ‐CacheGROUPBYExampleApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

InteractiveSQLwithApacheZeppelinApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.

eigniteApache ,ApacheIgnite,Ignite itedStatesand/orothercountries.@dsetrakyan

Apache ,)Apache)Ignite,)Ignite oundation)in)the)United)States .