apache-ignite

1
// Licensed to the Apache Software Foundation (ASF) under one or more
2
// contributor license agreements.  See the NOTICE file distributed with
3
// this work for additional information regarding copyright ownership.
4
// The ASF licenses this file to You under the Apache License, Version 2.0
5
// (the "License"); you may not use this file except in compliance with
6
// the License.  You may obtain a copy of the License at
7
//
8
// http://www.apache.org/licenses/LICENSE-2.0
9
//
10
// Unless required by applicable law or agreed to in writing, software
11
// distributed under the License is distributed on an "AS IS" BASIS,
12
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
// See the License for the specific language governing permissions and
14
// limitations under the License.
15
= Network Configuration
16
:javaFile: {javaCodeDir}/NetworkConfiguration.java
17
:xmlFile: code-snippets/xml/network-configuration.xml
18

19
== IPv4 vs IPv6
20

21
Ignite tries to support IPv4 and IPv6 but this can sometimes lead to issues where the cluster becomes detached. A possible solution — unless you require IPv6 — is to restrict Ignite to IPv4 by setting the `-Djava.net.preferIPv4Stack=true` JVM parameter.
22

23

24
== Discovery
25
This section describes the network parameters of the default discovery mechanism, which uses the TCP/IP protocol to exahcange discovery messages and is implemented in the `TcpDiscoverySpi` class.
26

27
You can change the properties of the discovery mechanism as follows:
28

29
[tabs]
30
--
31
tab:XML[]
32
[source, xml]
33
----
34
include::{xmlFile}[tags=!*;ignite-config;discovery, indent=0]
35
----
36
tab:Java[]
37
[source, java]
38
----
39
include::{javaFile}[tags=discovery, indent=0]
40

41
----
42

43
tab:C#/.NET[]
44

45
tab:C++[unsupported]
46

47
--
48

49
The following table describes some most important properties of `TcpDiscoverySpi`.
50
You can find the complete list of properties in the javadoc:org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi[] javadoc.
51

52
[CAUTION]
53
====
54
You should initialize the `IgniteConfiguration.localHost` or `TcpDiscoverySpi.localAddress` parameter with the network
55
interface that will be used for inter-node communication. By default, a node binds to and listens on all available IP
56
addresses of an environment it's running on. It can prolong node failures detection if some of the node's addresses are
57
not reachable from other cluster nodes.
58
====
59

60
[cols="1,2,1",opts="header"]
61
|===
62
|Property | Description| Default Value
63
| `localAddress`| Local host IP address used for discovery. If set, overrides the `IgniteConfiguration.localHost` setting. | By default, a node binds to all available network addresses. If there is a non-loopback address available, then java.net.InetAddress.getLocalHost() is used.
64
| `localPort`  | The port that the node binds to. If set to a non-default value, other cluster nodes must know this port to be able to discover the node. | `47500`
65
| `localPortRange`| If the `localPort` is busy, the node attempts to bind to the next port (incremented by 1) and continues this process until it finds a free port. The `localPortRange` property defines the number of ports the node will try (starting from `localPort`).
66
   | `100`
67
| `soLinger`| Specifies a linger-on-close timeout of TCP sockets used by Discovery SPI. See Java `Socket.setSoLinger` API
68
for details on how to adjust this setting. In Ignite, the timeout defaults to a non-negative value to prevent
69
link:https://bugs.openjdk.java.net/browse/JDK-8219658[potential deadlocks with SSL connections, window=_blank] but,
70
as a side effect, this can prolong the detection of cluster node failures. Alternatively, update your JRE version to the
71
one with the SSL issue fixed and adjust this setting accordingly. | `0`
72
| `reconnectCount` | The number of times the node tries to (re)establish connection to another node. |`10`
73
| `networkTimeout` |  The maximum network timeout in milliseconds for network operations. |`5000`
74
| `socketTimeout` |  The socket operations timeout. This timeout is used to limit connection time and write-to-socket time. |`5000`
75
| `ackTimeout`| The acknowledgement timeout for discovery messages.
76
If an acknowledgement is not received within this timeout, the discovery SPI tries to resend the message.  |  `5000`
77
| `joinTimeout` |  The join timeout defines how much time the node waits to join a cluster. If a non-shared IP finder is used and the node fails to connect to any address from the IP finder, the node keeps trying to join within this timeout. If all addresses are unresponsive, an exception is thrown and the node terminates.
78
`0` means waiting indefinitely.  | `0`
79
| `statisticsPrintFrequency` | Defines how often the node prints discovery statistics to the log.
80
`0` indicates no printing. If the value is greater than 0, and quiet mode is disabled, then statistics is printed out at INFO level once every period. | `0`
81

82
|===
83

84

85

86
== Communication
87

88
After the nodes discover each other and the cluster is formed, the nodes exchange messages via the communication SPI.
89
The messages represent distributed cluster operations, such as task execution, data modification operations, queries, etc.
90
The default implementation of the communication SPI uses the TCP/IP protocol to exchange messages (`TcpCommunicationSpi`).
91
This section describes the properties of `TcpCommunicationSpi`.
92

93
Each node opens a local communication port and address to which other nodes connect and send messages.
94
At startup, the node tries to bind to the specified communication port (default is 47100).
95
If the port is already used, the node increments the port number until it finds a free port.
96
The number of attempts is defined by the `localPortRange` property (defaults to 100).
97

98
[tabs]
99
--
100
tab:XML[]
101
[source, xml]
102
----
103
include::{xmlFile}[tags=!*;ignite-config;communication-spi, indent=0]
104
----
105

106
tab:Java[]
107
[source, java]
108
----
109
include::{javaCodeDir}/ClusteringOverview.java[tag=commSpi,indent=0]
110
----
111

112
tab:C#/.NET[]
113
[source,csharp]
114
----
115
include::code-snippets/dotnet/ClusteringOverview.cs[tag=CommunicationSPI,indent=0]
116
----
117
tab:C++[unsupported]
118
--
119

120
Below is a list of some important properties of `TcpCommunicationSpi`.
121
You can find the list of all properties in the javadoc:org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi[] javadoc.
122

123
[cols="1,2,1",opts="header"]
124
|===
125
|Property | Description| Default Value
126
| `localAddress` | The local address for the communication SPI to bind to. If set, overrides the `IgniteConfiguration.localHost` setting. |
127

128
| `localPort` | The local port that the node uses for communication.  | `47100`
129

130
| `localPortRange` | The range of ports the nodes tries to bind to sequentially until it finds a free one. |  `100`
131

132
|`tcpNoDelay` | Sets the value for the `TCP_NODELAY` socket option. Each socket accepted or created will use the provided value.
133

134
The option should be set to `true` (default) to reduce request/response time during communication over TCP. In most cases we do not recommend changing this option.| `true`
135

136
|`idleConnectionTimeout` | The maximum idle connection timeout (in milliseconds) after which the connection is closed. |  `600000`
137

138
|`usePairedConnections` | Whether dual socket connection between the nodes should be enforced. If set to `true`, two separate connections will be established between the communicating nodes: one for outgoing messages, and one for incoming messages. When set to `false`, a single TCP connection will be used for both directions.
139
This flag is useful on some operating systems when messages take too long to be delivered.   | `false`
140

141
| `directBuffer` | A boolean flag that indicates whether to allocate NIO direct buffer instead of NIO heap allocation buffer. Although direct buffers perform better, in some cases (especially on Windows) they may cause JVM crashes. If that happens in your environment, set this property to `false`.   | `true`
142

143
|`directSendBuffer` | Whether to use NIO direct buffer instead of NIO heap allocation buffer when sending messages.   | `false`
144

145
|`socketReceiveBuffer`| Receive buffer size for sockets created or accepted by the communication SPI. If set to `0`,   the operating system's default value is used. | `0`
146

147
|`socketSendBuffer` | Send buffer size for sockets created or accepted by the communication SPI. If set to `0` the  operating system's default value is used. | `0`
148

149
|===
150

151

152
== Connection Timeouts
153

154
////
155
//Connection timeout is a period of time a cluster node waits before a connection to another node is considered "failed".
156

157
Every node in a cluster is connected to every other node.
158
When node A sends a message to node B, and node B does not reply in `failureDetectionTimeout` (in milliseconds), then node B will be removed from the cluster.
159
////
160

161
There are several properties that define connection timeouts:
162

163
[cols="",opts="header"]
164
|===
165
|Property | Description | Default Value
166
| `IgniteConfiguration.failureDetectionTimeout` | A timeout for basic network operations for server nodes. | `10000`
167

168
| `IgniteConfiguration.clientFailureDetectionTimeout` | A timeout for basic network operations for client nodes.  | `30000`
169

170
|===
171

172
//CAUTION: The timeout automatically controls configuration parameters of `TcpDiscoverySpi`, such as socket timeout, message acknowledgment timeout and others. If any of these parameters is set explicitly, then the failure timeout setting will be ignored.
173

174
:ths:  
175

176
You can set the failure detection timeout in the node configuration as shown in the example below.
177
//The default value is 10{ths}000 ms for server nodes and 30{ths}000 ms for client nodes.
178
The default values allow the discovery SPI to work reliably on most on-premise and containerized deployments.
179
However, in stable low-latency networks, you can set the parameter to {tilde}200 milliseconds in order to detect and react to​ failures more quickly.
180

181
[tabs]
182
--
183
tab:XML[]
184
[source,xml]
185
----
186
include::code-snippets/xml/network-configuration.xml[tags=!*;ignite-config;failure-detection-timeout, indent=0]
187
----
188

189
tab:Java[]
190
[source,java]
191
----
192
include::{javaFile}[tag=failure-detection-timeout, indent=0]
193
----
194
tab:C#/.NET[]
195

196
tab:C++[unsupported]
197

198
--
199