1
<#import "/templates/guide.adoc" as tmpl>
2
<#import "/templates/kc.adoc" as kc>
3
<#import "/templates/options.adoc" as opts>
4
<#import "/templates/links.adoc" as links>
7
title="Configuring distributed caches"
8
summary="Understand how to configure the caching layer"
9
includedOptions="cache cache-*">
11
{project_name} is designed for high availability and multi-node clustered setups.12
The current distributed cache implementation is built on top of https://infinispan.org[Infinispan], a high-performance, distributable in-memory data grid.
14
== Enable distributed caching
15
When you start {project_name} in production mode, by using the `start` command, caching is enabled and all {project_name} nodes in your network are discovered.17
By default, caches are using a UDP transport stack so that nodes are discovered using IP multicast transport based on UDP. For most production environments, there are better discovery alternatives to UDP available. {project_name} allows you to either choose from a set of pre-defined default transport stacks, or to define your own custom stack, as you will see later in this {section}.19
To explicitly enable distributed infinispan caching, enter this command:
21
<@kc.build parameters="--cache=ispn"/>
23
When you start {project_name} in development mode, by using the `start-dev` command, {project_name} uses only local caches and distributed caches are completely disabled by implicitly setting the `--cache=local` option.24
The `local` cache mode is intended only for development and testing purposes.
27
{project_name} provides a cache configuration file with sensible defaults located at `conf/cache-ispn.xml`.29
The cache configuration is a regular https://infinispan.org/docs/stable/titles/configuring/configuring.html[Infinispan configuration file].
31
The following table gives an overview of the specific caches {project_name} uses.32
You configure these caches in `conf/cache-ispn.xml`:
36
|Cache name|Cache Type|Description
38
|realms|Local|Cache persisted realm data
39
|users|Local|Cache persisted user data
40
|authorization|Local|Cache persisted authorization data
41
|keys|Local|Cache external public keys
42
|work|Replicated|Propagate invalidation messages across nodes
43
|authenticationSessions|Distributed|Caches authentication sessions, created/destroyed/expired during the authentication process
44
|sessions|Distributed|Caches user sessions, created upon successful authentication and destroyed during logout, token revocation, or due to expiration
45
|clientSessions|Distributed|Caches client sessions, created upon successful authentication to a specific client and destroyed during logout, token revocation, or due to expiration
46
|offlineSessions|Distributed|Caches offline user sessions, created upon successful authentication and destroyed during logout, token revocation, or due to expiration
47
|offlineClientSessions|Distributed|Caches client sessions, created upon successful authentication to a specific client and destroyed during logout, token revocation, or due to expiration
48
|loginFailures|Distributed|keep track of failed logins, fraud detection
49
|actionTokens|Distributed|Caches action Tokens
52
=== Cache types and defaults
55
{project_name} caches persistent data locally to avoid unnecessary round-trips to the database.57
The following data is kept local to each node in the cluster using local caches:
59
* *realms* and related data like clients, roles, and groups.
60
* *users* and related data like granted roles and group memberships.
61
* *authorization* and related data like resources, permissions, and policies.
64
Local caches for realms, users, and authorization are configured to hold up to 10,000 entries per default.
65
The local key cache can hold up to 1,000 entries per default and defaults to expire every one hour.
66
Therefore, keys are forced to be periodically downloaded from external clients or identity providers.
68
In order to achieve an optimal runtime and avoid additional round-trips to the database you should consider looking at
69
the configuration for each cache to make sure the maximum number of entries is aligned with the size of your database. More entries
70
you can cache, less often the server needs to fetch data from the database. You should evaluate the trade-offs between memory utilization and performance.
72
.Invalidation of local caches
73
Local caching improves performance, but adds a challenge in multi-node setups.
75
When one {project_name} node updates data in the shared database, all other nodes need to be aware of it, so they invalidate that data from their caches.77
The `work` cache is a replicated cache and used for sending these invalidation messages. The entries/messages in this cache are very short-lived,
78
and you should not expect this cache growing in size over time.
80
.Authentication sessions
81
Authentication sessions are created whenever a user tries to authenticate. They are automatically destroyed once the authentication process
82
completes or due to reaching their expiration time.
84
The `authenticationSessions` distributed cache is used to store authentication sessions and any other data associated with it
85
during the authentication process.
87
By relying on a distributable cache, authentication sessions are available to any node in the cluster so that users can be redirected
88
to any node without losing their authentication state. However, production-ready deployments should always consider session affinity and favor redirecting users
89
to the node where their sessions were initially created. By doing that, you are going to avoid unnecessary state transfer between nodes and improve
90
CPU, memory, and network utilization.
94
Once the user is authenticated, a user session is created. The user session tracks your active users and their state so that they can seamlessly
95
authenticate to any application without being asked for their credentials again. For each application, the user authenticates with a client session
96
is created too, so that the server can track the applications the user is authenticated with and their state on a per-application basis.
98
User and client sessions are automatically destroyed whenever the user performs a logout, the client performs a token revocation, or due to reaching their expiration time.
100
The following caches are used to store both user and client sessions:
105
By relying on a distributable cache, user and client sessions are available to any node in the cluster so that users can be redirected
106
to any node without loosing their state. However, production-ready deployments should always consider session affinity and favor redirecting users
107
to the node where their sessions were initially created. By doing that, you are going to avoid unnecessary state transfer between nodes and improve
108
CPU, memory, and network utilization.
110
As an OpenID Connect Provider, the server is also capable of authenticating users and issuing offline tokens. Similarly to regular user and client sessions,
111
when an offline token is issued by the server upon successful authentication, the server also creates an offline user session and an offline client session. However, due to the nature
112
of offline tokens, offline sessions are handled differently as they are long-lived and should survive a complete cluster shutdown. Because of that, they are also persisted to the database.
114
The following caches are used to store offline sessions:
117
* offlineClientSessions
119
Upon a cluster restart, offline sessions are lazily loaded from the database and kept in a shared cache using the two caches above.
121
.Password brute force detection
122
The `loginFailures` distributed cache is used to track data about failed login attempts.
123
This cache is needed for the Brute Force Protection feature to work in a multi-node {project_name} setup.126
Action tokens are used for scenarios when a user needs to confirm an action asynchronously, for example in the emails sent by the forgot password flow.
127
The `actionTokens` distributed cache is used to track metadata about action tokens.
129
=== Configuring caches for availability
131
Distributed caches replicate cache entries on a subset of nodes in a cluster and assigns entries to fixed owner nodes.
133
Each distributed cache has two owners per default, which means that two nodes have a copy of the specific cache entries.
134
Non-owner nodes query the owners of a specific cache to obtain data.
135
When both owner nodes are offline, all data is lost.
136
This situation usually leads to users being logged out at the next request and having to log in again.
138
The default number of owners is enough to survive 1 node (owner) failure in a cluster setup with at least three nodes. You are free
139
to change the number of owners accordingly to better fit into your availability requirements. To change the number of owners, open `conf/cache-ispn.xml` and change the value for `owners=<value>` for the distributed caches to your desired value.
141
=== Specify your own cache configuration file
143
To specify your own cache configuration file, enter this command:
145
<@kc.build parameters="--cache-config-file=my-cache-file.xml"/>
147
The configuration file is relative to the `conf/` directory.
149
=== CLI options for remote server
151
For configuration of {project_name} server for high availability and multi-node clustered setup there was introduced following CLI options `cache-remote-host`, `cache-remote-port`, `cache-remote-username` and `cache-remote-password` simplifying configuration within the XML file.152
Once any of declared CLI parameters are present, it is expected there is no configuration related to remote store present in the XML file.
156
Transport stacks ensure that distributed cache nodes in a cluster communicate in a reliable fashion.
157
{project_name} supports a wide range of transport stacks:159
<@opts.expectedValues option="cache-stack"/>
161
To apply a specific cache stack, enter this command:
163
<@kc.build parameters="--cache-stack=<stack>"/>
165
The default stack is set to `udp` when distributed caches are enabled.
167
=== Available transport stacks
169
The following table shows transport stacks that are available without any further configuration than using the `--cache-stack` build option:
173
|Stack name|Transport protocol|Discovery
175
|tcp|TCP|MPING (uses UDP multicast).
176
|udp|UDP|UDP multicast
179
The following table shows transport stacks that are available using the `--cache-stack` build option and a minimum configuration:
183
|Stack name|Transport protocol|Discovery
185
|kubernetes|TCP|DNS_PING (requires `-Djgroups.dns.query=<headless-service-FQDN>` to be added to JAVA_OPTS or JAVA_OPTS_APPEND environment variable).
188
=== Additional transport stacks
189
The following table shows transport stacks that are supported by {project_name}, but need some extra steps to work.190
Note that _none_ of these stacks are Kubernetes / OpenShift stacks, so no need exists to enable the `google` stack if you want to run {project_name} on top of the Google Kubernetes engine.191
In that case, use the `kubernetes` stack.
192
Instead, when you have a distributed cache setup running on AWS EC2 instances, you would need to set the stack to `ec2`, because ec2 does not support a default discovery mechanism such as UDP.
196
|Stack name|Transport protocol|Discovery
198
|ec2|TCP|NATIVE_S3_PING
199
|google|TCP|GOOGLE_PING2
203
Cloud vendor specific stacks have additional dependencies for {project_name}.204
For more information and links to repositories with these dependencies, see the https://infinispan.org/docs/dev/titles/embedding/embedding.html#jgroups-cloud-discovery-protocols_cluster-transport[Infinispan documentation].
206
To provide the dependencies to {project_name}, put the respective JAR in the `providers` directory and build Keycloak by entering this command:208
<@kc.build parameters="--cache-stack=<ec2|google|azure>"/>
210
=== Custom transport stacks
211
If none of the available transport stacks are enough for your deployment, you are able to change your cache configuration file
212
and define your own transport stack.
214
For more details, see https://infinispan.org/docs/stable/titles/server/server.html#using-inline-jgroups-stacks_cluster-transport[Using inline JGroups stacks].
216
.defining a custom transport stack
220
<stack name="my-encrypt-udp" extends="udp">
221
<SSL_KEY_EXCHANGE keystore_name="server.jks"
222
keystore_password="password"
223
stack.combine="INSERT_AFTER"
224
stack.position="VERIFY_SUSPECT2"/>
225
<ASYM_ENCRYPT asym_keylength="2048"
227
change_key_on_coord_leave = "false"
228
change_key_on_leave = "false"
229
use_external_key_exchange = "true"
230
stack.combine="INSERT_BEFORE"
231
stack.position="pbcast.NAKACK2"/>
235
<cache-container name="keycloak">
236
<transport lock-timeout="60000" stack="my-encrypt-udp"/>
241
By default, the value set to the `cache-stack` option has precedence over the transport stack you define in the cache configuration file.
242
If you are defining a custom stack, make sure the `cache-stack` option is not used for the custom changes to take effect.
244
== Securing cache communication
245
The current Infinispan cache implementation should be secured by various security measures such as RBAC, ACLs, and transport stack encryption.
247
JGroups handles all the communication between {project_name} server, and it supports Java SSL sockets for TCP communication.248
{project_name} uses CLI options to configure the TLS communication without having to create a customized JGroups stack or modifying the cache XML file.250
To enable TLS, `cache-embedded-mtls-enabled` must be set to `true`.
251
It requires a keystore with the certificate to use: `cache-embedded-mtls-key-store-file` sets the path to the keystore, and `cache-embedded-mtls-key-store-password` sets the password to decrypt it.
252
The truststore contains the valid certificates to accept connection from, and it can be configured with `cache-embedded-mtls-trust-store-file` (path to the truststore), and `cache-embedded-mtls-trust-store-password` (password to decrypt it).
253
To restrict unauthorized access, use a self-signed certificate for each {project_name} deployment.255
For JGroups stacks with `UDP` or `TCP_NIO2`, see the http://jgroups.org/manual5/index.html#ENCRYPT[JGroups Encryption documentation] on how to set up the protocol stack.
257
For more information about securing cache communication, see the https://infinispan.org/docs/stable/titles/security/security.html#[Infinispan security guide].
259
== Exposing metrics from caches
261
By default, metrics from caches are not automatically exposed when the metrics are enabled.
262
For more details about how to enable metrics, see <@links.server id="configuration-metrics"/>.
264
To enable global metrics for all caches within the `cache-container`, you need to change your cache configuration file (e.g.: `conf/cache-ispn.xml`) to enable `statistics` at the `cache-container` level as follows:
266
.enabling metrics for all caches
269
<cache-container name="keycloak" statistics="true">
274
Similarly, you can enable metrics individually for each cache by enabling `statistics` as follows:
276
.enabling metrics for a specific cache
278
<local-cache name="realms" statistics="true">