Transcription
Tomcat ClusterKeiichi Fujino
Agenda About meTomcat Clustering OverviewSession ReplicationCluster Channel ComponentMonitoring Cluster components
About me Keiichi FujinoI live in JapanSoftware Enginner since 2002Apache Tomcat committer since 2010 kfujino@apache.org
Clustering OverviewWhat is Cluster? Performance improvement High availabilityTomcat Clustering Cluster membership/Grouping Session ReplicationLoad balancing is not a Tomcat features Use mod jk / mod proxy balancer
Cluster loyerCreate fromtemplateManagerContext
Cluster Architecture Cluster The main component of Tomcat Cluster Cluster Manager The session manager for the session replication Valve(Cluster Valve) The same as usual Valve Added to the request processing pipelineautomatically
Cluster Architecture Cluster Deployer Sharing of WAR files among cluster nodes ClusterListener Listen cluster messages and events Channel Performs messaging and grouping among thecluster nodes
Session ReplicationSession Replication
Session ReplicationImplementations All-to-All session replication DeltaManager (Default)Tomcat1Replicate to all nodesTomcat3 Primary-Secondary sessionreplication condaryReplicate to backup nodeTomcat3Tomcat4
Use constraints Make sure that your web.xml describe the distributable/ element Session attributes must implement java.io.Serializable sticky session If you use the BackupManager, This is required
How to configure Define Cluster element inside The Engine or Host element. Configure Cluster Manager DeltaManager or BackupManager Configure Channel components Enable org.apache.catalina.ha.tcp.ReplicationValve Enable ner If DeltaManager use
Delta ReplicationDelta Replication Replicate only the changes of session Not all session data Replicate all changes of session at the time of end of request Not replication per change of session
Delta ReplicationRegister Delta Info Add ATTRIBUTE(Attr A, Value A)Add ATTRIBUTE(Attr B, Value B)Remove ATTRIBUTE(Attr A)Add ATTRIBUTE(Attr B, Value BB)Default : recordAllActions falserecordAllActions ESETAttr AValue AATTRIBUTESETAttr AValue AATTRIBUTESETAttr BValue BATTRIBUTESETAttr BValue BATTRIBUTEREMOVEAttr AnullATTRIBUTEREMOVEAttr AnullATTRIBUTESETAttr BValue BBATTRIBUTESETAttr BValue BB
DeltaManagerDeltaManager All-to-All session replication Default Session Manager in Cluster environment For small cluster
tionClusterchannelClusterSessionListenerReceive replication MessageDelta replicationDeltaManager
Session(Non-Primary)Replicate ry)Session(Non-Primary)
DeltaManagerNode FailureTomcat1Node ary)
DeltaManagerNode RecoveryGet All session MessageTomcat1No Session All Session Data MessageSession(Pri)Complete ion(Non-Pri)Session(Pri)Session(Non-Pri)
BackupManagerBackupManager Primary-Secondary session replication For large cluster
ValveDelta channel
BackupManagerBehavior of CreateTomcat1Create sessionReplicate session e )
BackupManagerBehavior of CreateTomcat1Tomcat2Session (Primary)Session (Backup)Session (Proxy)Session (Proxy)Replicate sessionIdCreate sessionTomcat3Session (Proxy)Session (Primary)Replicate session dataTomcat4Session (Proxy)Session (Backup)
BackupManagerBehaviorof DeltaTomcat2Tomcat1Replicate Session(Proxy)Session(Proxy)
BackupManagerBehavior of RemoveTomcat1Remove e sessionTomcat3Tomcat4Session(Proxy)Session(Proxy)
BackupManagerNode FailureTomcat2Tomcat1Promote&Select new backup&publishSessionA(Primary)Node (Backup)SessionB(Primary)Select new backup&publishSessionC(Proxy)NOPTomcat4newly backupnewly proxySessionC(Primary)NOPnewly )SessionB(Proxy) newly backup SessionB(Backup)SessionC(Backup)NOPSessionA(Proxy)
BackupManagerNode RecoveryTomcat1No Session ive RPC Message INIT Message STATE Message START e RPC Message INIT Message START y)SessionB(Primary)SessionB(Backup)
ChannelCluster Channel
ChannelWhat is Channel ? Messaging & Grouping componentResponsibility Build Membership Send channel messages Receive channel messages
Channel elInterceptorChannelCoordinator
Channel Components Channel The main component of el only MembershipService The component which build a cluster groupStart a multicast receivier thread and a multicast senderthread
Channel Components ChannelSender Send channel messages to other nodesSender Queue size is specified in poolSize attributeChannelReceiver Receive channel messages from other nodesTuning of the maxThreads attribute depends on send option ofchannel message Synchronization mode Need to align maxThreads with poolSizeAsynchronous mode do not need to align the maxThreads with poolSize.
Channel Components ChannelListener Listen received channel messagesMembershipListener Listen add/remove of cluster members
Channel Components ChannelInterceptor Intercept a channel message and a member detectionThere are many configurable implementationsChannelCoordinator Special ChannelInterceptorCoordinates the ChannelInterceptors
Sample config Cluster className "org.apache.catalina.ha.tcp.SimpleTcpCluster" ・・・ ・・・ Channel className "org.apache.catalina.tribes.group.GroupChannel" Membership className e"address "229.0.0.61“ port "45564“ frequency "500“ dropTime "4000" / Receiver className ver"address “auto“ port ”4004“ autoBind ”100” maxThreads “25”/ Sender className ransmitter" Transport className rallelSender"timeout "5000“ poolSize "25"/ /Sender Interceptor className sageDispatch15Interceptor" / Interceptor className FailureDetector"/ /Channel ・・・ ・・・ /Cluster
Channel ArchitectureTomcat1ChannelListenerNotify message receiveMembershipListenerChannelNotify member receiveChannelInterceptor(1)Send MessageReceive Message/Member ershipServiceMembership MessagingChannelSenderSend MessageChannelReceiverReceive Message
ChannelInterceptorTCPFailureDetector Main Features Intercept memberDisappeared events. Member check when send errors. Other Features Manage membership in static membership
ChannelInterceptorMessageDispatchInterceptor Asynchronous send message Use Thread Pooling maxThreads maxSpareThreads You must set the send options asynchronous. Cluster’s channelSendOptions to asyn mode(8) BackupManager’s mapSendOptions to asyn mode(8)
ChannelInterceptorStaticMembershipInterceptor The static membership instead of multicast Make sure Disable multicast membership Cluster’s channelStartOptions attribute 3 Enable TcpPingInterceptor for nodes failure detection Enable TcpFailureDetector for membership management The order shipInterceptor
ChannelInterceptor Interceptor className PingInterceptor"/ Interceptor className FailureDetector"/ Interceptor className ticMembershipInterceptor" LocalMember className r"uniqueId "{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,1}"/ Member className r"port "4010" host "hostA"uniqueId "{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,2}" / Member className r"port "4010" host "hostB"uniqueId "{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,3}" / /Interceptor
ChannelInterceptorThroughputInterceptor Measuring the send and receive of channel messagesDomainFilterInterceptor Filter the members that join cluster group by the domain
Monitoring ClusterMonitoring Cluster
Monitoring ClusterHow to monitor Tomcat cluster. Logging JMX
Monitoring ClusterMonitoring your Cluster with Logging To track channel messages for Debugging Use the Tomcat JULI Key: org.apache.catalina.tribes.MESSAGES Level: FINEST# FOR ormatter ache.juli.AsyncFileHandler.level directory ncFileHandler.prefix er.bufferSize -1org.apache.catalina.tribes.MESSAGES.level FINESTorg.apache.catalina.tribes.MESSAGES.handlers 10catalina.org.apache.juli.AsyncFileHandler
Monitoring ClusterMonitoring your Cluster with Logging ThroughputInterceptor Report the throughput statistics. Default interval is to report every 10000 messages. Key: ughputInterceptor Level: ThroughputInterceptor.report ThroughputInterceptor Report[Tx Msg:60,024 messagesSent:53.30 MB (total)Sent:53.31 MB (application)Time:16.84 secondsTx Speed:3.16 MB/sec (total)TxSpeed:3.16 MB/sec (application)Error Msg:0Rx Msg:60,048 messagesRx Speed:0.23 MB/sec (since 1st msg)Received:53.28 MB]
Monitoring ClusterMonitoring your Cluster with JMX Catalina Domain Cluster Mbeans Cluster Mbean Deployer Mbean Member MBeans (Cluster )Manager Mbean (Cluster )Valve MBean
Monitoring ClusterCluster Mbean Cluster settings
Monitoring ClusterMember Mbeans All Cluster members that have been joining same cluster group
Monitoring Cluster(Cluster )Manager Mbean Session Information
Monitoring Cluster(Cluster )Valve MBean ReplicationValve settings
Monitoring ClusterMonitoring your Cluster with JMX ClusterChannel Domain Channel Mbeans Channel Mbean Interceptor Mbeans Membership Mbean Receiver Mbean Sender MBean
Monitoring ClusterChannel Mbean Channel settings
Monitoring ClusterMembership Mbean Membership Settings
Monitoring ClusterMembership Mbean Membership Operations
Monitoring ClusterSender MBean Settings and Stats info
Monitoring ClusterReceiver Mbean Settings and Stats
Monitoring ClusterInterceptor Mbeans All interceptor that are used in channel Implements MBeans of all commonly usedInterceptors but It does not implement allInterceptor MBeans.
Monitoring ClusterThroughputInterceptor The throughput statistics
Monitoring ClusterTcpFailureDetector Settings and Member check by TCP
Monitoring ClusterMessageDispatchInterceptor Settings and Stats info
Monitoring ClusterIt's important to note that Channel Mbeans are supported in 9.0.0.M20 and later This feature has not back-ported into Tomcat8.5 yet. The ReplicationMap MBean is T.B.D
Questions?
Thank You
Tomcat ClusterKeiichi Fujino
Tomcat Clustering Overview Session Replication Cluster Channel Component Monitoring Cluster components. About me Keiichi Fujino . Monitoring your Cluster with JMX ClusterChannel Domain Channel Mbeans
