apache-ignite

1
// Licensed to the Apache Software Foundation (ASF) under one or more
2
// contributor license agreements.  See the NOTICE file distributed with
3
// this work for additional information regarding copyright ownership.
4
// The ASF licenses this file to You under the Apache License, Version 2.0
5
// (the "License"); you may not use this file except in compliance with
6
// the License.  You may obtain a copy of the License at
7
//
8
// http://www.apache.org/licenses/LICENSE-2.0
9
//
10
// Unless required by applicable law or agreed to in writing, software
11
// distributed under the License is distributed on an "AS IS" BASIS,
12
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
// See the License for the specific language governing permissions and
14
// limitations under the License.
15
= ZooKeeper Discovery
16

17
Ignite's default TCP/IP Discovery organizes cluster nodes into a ring topology that has advantages and
18
disadvantages. For instance, on topologies with hundreds of cluster
19
nodes, it can take many seconds for a system message to traverse through
20
all the nodes. As a result, the basic processing of events such as
21
joining of new nodes or detecting the failed ones can take a while,
22
affecting the overall cluster responsiveness and performance.
23

24
ZooKeeper Discovery is designed for massive deployments that
25
need to preserve ease of scalability and linear performance.
26
However, using both Ignite and ZooKeeper requires configuring and managing two
27
distributed systems, which can be challenging.
28
Therefore, we recommend that you use ZooKeeper Discovery only if you plan to scale to 100s or 1000s nodes.
29
Otherwise, it is best to use link:clustering/tcp-ip-discovery[TCP/IP Discovery].
30

31
ZooKeeper Discovery uses ZooKeeper as a single point of synchronization
32
and to organize the cluster into a star-shaped topology where a
33
ZooKeeper cluster sits in the center and the Ignite nodes exchange
34
discovery events through it.
35

36
image::images/zookeeper.png[Zookeeper]
37

38
It is worth mentioning that ZooKeeper Discovery is an alternative implementation of the Discovery SPI and doesn’t affect the Communication SPI.
39
Once the nodes discover each other via ZooKeeper Discovery, they use Communication SPI for peer-to-peer communication.
40
////////////////////////////////////////////////////////////////////////////////
41
TODO: explain what it means
42
////////////////////////////////////////////////////////////////////////////////
43

44
== Configuration
45

46
To enable ZooKeeper Discovery, you need to configure `ZookeeperDiscoverySpi` in a way similar to this:
47

48

49
[tabs]
50
--
51
tab:XML[]
52
[source,xml]
53
----
54
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
55

56
  <property name="discoverySpi">
57
    <bean class="org.apache.ignite.spi.discovery.zk.ZookeeperDiscoverySpi">
58
      <property name="zkConnectionString" value="127.0.0.1:34076,127.0.0.1:43310,127.0.0.1:36745"/>
59
      <property name="sessionTimeout" value="30000"/>
60
      <property name="zkRootPath" value="/apacheIgnite"/>
61
      <property name="joinTimeout" value="10000"/>
62
    </bean>
63
  </property>
64
</bean>
65
----
66
tab:Java[]
67
[source,java]
68
----
69
include::{javaCodeDir}/ZookeeperDiscovery.java[tag=cfg,indent=0]
70
----
71
tab:.NET[unsupported]
72
tab:C++[unsupported]
73
--
74

75
The following parameters are required (other parameters are optional):
76

77
* `zkConnectionString` - keeps the list of addresses of ZooKeeper
78
servers.
79
* `sessionTimeout` - specifies the time after which an Ignite node is considered disconnected if it doesn’t react to events exchanged via Discovery SPI.​
80

81
== Failures and Split Brain Handling
82

83
In case of network partitioning, some of ​the nodes cannot communicate to each other because they are located in separated network segments, which may lead to failure to process user requests or inconsistent data modification.
84

85
ZooKeeper Discovery approaches network partitioning (aka. split brain)
86
and communication failures between individual nodes in the following
87
way:
88

89
[CAUTION]
90
====
91
It is assumed that the ZooKeeper cluster is always visible to all the
92
nodes in the cluster. In fact, if a node disconnects from ZooKeeper, it
93
shuts down and other nodes treat it as failed or disconnected.
94
====
95

96
Whenever a node discovers that it cannot connect to some of the other
97
nodes in the cluster, it initiates a communication failure resolution
98
process by publishing special requests to the ZooKeeper cluster. When
99
the process is started, all nodes try to connect to each other and send
100
the results of the connection attempts to the node that coordinates the
101
process (_the coordinator node_). Based on this information, the
102
coordinator node creates a connectivity graph that represents the
103
network situation in the cluster. Further actions depend on the type of
104
network segmentation. The following sections discuss possible scenarios.
105

106
=== Cluster is split into several disjoint components
107

108
If the cluster is split into several independent components, each
109
component (being a cluster) may think of itself as a master cluster and
110
continue to process user requests, resulting in data inconsistency. To
111
avoid this, only the component with the largest number of nodes is kept
112
alive; and the nodes from the other components are brought down.
113

114
image::images/network_segmentation.png[Network Segmentation]
115

116
The image above shows a case where the cluster network is split into 2 segments.
117
The nodes from the smaller cluster (right-hand segment) are terminated.
118

119
image::images/segmentation_resolved.png[Segmentation Resolved]
120

121
When there are multiple largest components, the one that has the largest
122
number of clients is kept alive, and the others are shut down.
123

124
=== Several links between nodes are missing
125

126
Some nodes cannot connect to some other nodes, which means the nodes are
127
not completely disconnected from the cluster but can’t exchange data
128
with some of the nodes and, therefore, cannot be part of the cluster. In
129
the image below, one node cannot connect to two other nodes.
130

131
image::images/split_brain.png[Split-brain]
132

133
In this case, the task is to find the largest component in which every
134
node can connect to every other node, which, in the general case, is a
135
difficult problem and cannot be solved in an acceptable amount of time. The
136
coordinator node uses a heuristic algorithm to find the best approximate
137
solution. The nodes that are left out of the solution are shut down.
138

139
image::images/split_brain_resolved.png[Split-brain Resolved]
140

141
=== ZooKeeper cluster segmentation
142

143
In large-scale deployments where the ZooKeeper cluster can span multiple data centers and geographically diverse locations, it can split into multiple segments due to network segmentation.
144
If this occurs, ZooKeeper checks if there is a segment that contains more than a half of all ZooKeeper nodes (ZooKeeper requires this many nodes to continue its operation), and, if found, this segment takes over managing the Ignite cluster, while other segments are shut down.
145
If there is no such segment, ZooKeeper shuts down all its nodes.
146

147
In case of ZooKeeper cluster segmentation, the Ignite cluster may or may not be split.
148
In any case, when the ZooKeeper nodes are shut down, the corresponding Ignite nodes try to connect to available ZooKeeper nodes and shut down if unable to do so.
149

150
The following image is an example of network segmentation that splits both the Ignite cluster and ZooKeeper cluster into two segments.
151
This may happen if your clusters are deployed in two data centers.
152
In this case, the ZooKeeper node located in Data Center B shuts itself down.
153
The Ignite nodes located in Data Center B are not able to connect to the remaining ZooKeeper nodes and shut themselves down as well.
154

155
image::images/zookeeper_split.png[Zookeeper Split]
156

157
== Custom Discovery Events
158

159
Changing a ring-shaped topology to the star-shaped one affects the way
160
custom discovery events are handled by the Discovery SPI component. Since
161
the ring topology is linear, it means that each discovery message is
162
processed by nodes sequentially.
163

164
With ZooKeeper Discovery, the coordinator sends discovery messages to
165
all nodes simultaneously resulting in the messages to be processed in
166
parallel. As a result, ZooKeeper Discovery prohibits custom discovery events from being changed. For instance, the nodes are not allowed to add any payload to discovery messages.
167

168
== Ignite and ZooKeeper Configuration Considerations
169

170
When using ZooKeeper Discovery, you need to make sure that the configuration parameters of the ZooKeeper cluster and Ignite cluster match each other.
171

172
Consider a sample ZooKeeper configuration, as follows:
173

174
[source,shell]
175
----
176
# The number of milliseconds of each tick
177
tickTime=2000
178

179
# The number of ticks that can pass between sending a request and getting an acknowledgement
180
syncLimit=5
181
----
182

183
Configured this way, ZooKeeper server detects its own segmentation from the rest of the ZooKeeper cluster only after `tickTime * syncLimit` elapses.
184
Until this event is detected at ZooKeeper level, all Ignite nodes connected to the segmented ZooKeeper server do not try to reconnect to the other ZooKeeper servers.
185

186
On the other hand, there is a `sessionTimeout` parameter on the Ignite
187
side that defines how soon ZooKeeper closes an Ignite node’s session if
188
the node gets disconnected from the ZooKeeper cluster.
189
If `sessionTimeout` is smaller than `tickTime * syncLimit` , then the
190
Ignite node is notified by the segmented ZooKeeper server too
191
late — its session expires before it tries to reconnect to other ZooKeeper servers.
192

193
To avoid this situation, `sessionTimeout` should be bigger than `tickTime * syncLimit`.
194