{docbook}
<abstract>
<para>
Clustering allows eXo Platfrom users to run various portal instances on several parallel servers which are also called nodes. The load is distributed across different servers, so the portal is still accessible via other cluster nodes in case of any failed servers.
Thanks to adding more nodes to the cluster, eXo Platform's performance can be much improved. In this chapter, you will see the following topics:
</para>
<itemizedlist>
<listitem>
<para>
<emphasis role="bold"><link linkend="ADM.Clustering.Cluster_setup">Set up eXo Platform cluster</link></emphasis>
</para>
<para>Information and characteristics of eXo Platform cluster, and steps to do a typical setup.</para>
</listitem>
<listitem>
<para>
<emphasis role="bold"><link linkend="ADM.Clustering.Clustering_FAQ">FAQs of clustering</link></emphasis>
</para>
<para>Common questions and answers that are useful for administrators when doing a clustering on eXo Platform.</para>
</listitem>
</itemizedlist>
</abstract>
{docbook}

A cluster is a set of nodes which are managed together and participate in the workload management. Installing eXo Platform in the cluster mode is considered in the following main cases:
* Load Balancing: when a single server node is not enough for handling the load.
* High Availability: if one of nodes is failed, the rest of nodes in the cluster can assume the workload of the system. It means that no access is interrupted.

These characteristics should be handled by the overall architecture of your system. The Load Balancing is typically achieved by a front server or device that distributes the request to the cluster nodes.
Also, the High Availability on the data layer can be typically achieved using the native replication implemented by Relation Database Management System (RDBMS) or Shared File Systems, such as SAN and NAS.

h1. Set up the eXo Platform cluster {anchor:id=ADM.Clustering.Cluster_setup}
{docbook}
<highlights>
 <itemizedlist>
 <listitem>
 <para>
 <link linkend="ADM.Clustering.Shared_file_system">Shared file system</link>
 </para>
 <para>
  Introduction to the shared file system which is necesary for the embedded JCR server.
 </para>
 </listitem>
 <listitem>
 <para>
  <link linkend="ADM.Clustering.Setup_Platform_cluster">Typical setup</link>
 </para>
 <para>
  Steps to do a typical setup of eXo Platform cluster.
</para>
 </listitem>
 <listitem>
 <para>
 <link linkend="ADM.Clustering.Advanced_configuration">Advanced configuration</link>
 </para>
 <para>
	Introduction to the advanced configuration which is optional and not required in general cases, including JBossCache and shared file system.
  </para>
 </listitem>
 <listitem>
 <para>
 <link linkend="ADM.Clustering.Local_JCR_Index">Local JCR index in cluster</link>
 </para>
 <para>
	Description of changes which will be applied if the cluster is used with the local JCR index on each node.
 </para>
 </listitem>
 </itemizedlist>
</highlights>
{docbook}

{docbook}
<para>
<prompt>
    <emphasis role="bold">See also</emphasis>
</prompt>
</para>
{docbook}
* [FAQs of clustering|#ADM.Clustering.Clustering_FAQ]

h2. Shared file system {anchor:id=ADM.Clustering.Shared_file_systemRoleNotInToc}

In eXo Platform, the persistence mostly relies on JCR, which is a middleware between the eXo Platform applications (including the Portal) and the database. Hence, this component must be configured to work in the cluster mode.

The embedded JCR server requires a portion of its state to be shared on a file system shared among the cluster nodes:
* The values storage.
* The index (in case of shared index usage).
{note}
Since Platform 3.5, a local JCR index can be used on each node of the cluster. It is a new feature and it needs a special configuration in Platform (described below). 

{note}

All nodes must have the read/write access to the shared file system.
{info}It is strongly recommended that you use a mount point on a SAN.{info}

h2. Typical setup {anchor:id=ADM.Clustering.Setup_Platform_clusterRoleNotInToc}
The following steps describe a typical setup of eXo Platform cluster:
* Switch to a _cluster_ configuration.
This step is done in the _configuration.properties_ file. This _configuration.properties_ file must be set in the same way on all the cluster nodes.
First, point the _exo.shared.dir_ variable to a directory shared between cluster nodes.
{code}
exo.shared.dir=/PATH/TO/SHARED/FS
{code}
The path is shared, so all nodes will need the _read/write_ access to this path.
Then, switch the JCR to the cluster mode.
{code}
gatein.jcr.config.type=cluster
{code}
In this step, JCR enables the automatic network replication and discovery between other cluster nodes.
* Switch to the _cluster_ profile.
You need to indicate the _cluster_ kernel profile to eXo Platform. This can be done by editing the startup script in the _bin/gatein.sh_ folder as below:
{code}
EXO_PROFILES="-Dexo.profiles=default,cluster"
{code}
or use the _start\_eXo.sh_ script with such parameters:
{code}
./start_eXo.sh default,cluster
{code}
* Do the initial startup.
For the initial startup of your JCR cluster, you should only start a single node. This node will initialize the internal JCR database and create the system workspace. Once the initial node is definitely started, you can start the other nodes.
{note}
This constraint is only for the initial start. 
As above, you can start the cluster in any order, but it should be started fully from the single node. After that, others can start in any order or in parallel.
{note}
* Start up and shut down.

{note}
It is recommended that you always start the cluster from a single node, as on initial startup, and then start all others in any order or in parallel.
Nodes of the cluster will automatically try to join others at startup. Once they have discovered each other, they will synchronize their state.
During the synchronization, the node is not ready to serve requests.
{note}

h2. Advanced configuration {anchor:id=ADM.Clustering.Advanced_configurationRoleNotInToc}

The cluster mode is preconfigured to work out of the box. The eXo Platform clustering fully relies on the JBossCache replication which  uses JGroups internally.
The default configuration of JBossCache lies in _exo.portal.component.common-x.x.x.jar_.
Since eXo Platform 3.5, the JCR's JBossCache configuration is externalized to the {{gatein.conf.dir}} configuration folder:
* _jcr_ - folder with cache configuration for JCR
* _cache_ - folder with cache configuration for eXo Cache 
* _idm_ - folder with cache configuration for PicketLink IDM organization service 
* _jgroups_ - folder with JGroups configuration used in all caches

{note}
The advanced configuration is optional and is not required in general cases. It is recommended to do an advanced configuration only in case of a need.
{note}

{docbook}
<para xml:id="ADM.Clustering.Advanced_configuration.JBossCache">
<citetitle>
<emphasis role="bold">
JBossCache
</emphasis>
</citetitle>
</para>
{docbook}

The JBossCache configuration is done in the _configuration.properties_ file using following properties:
{code}
# JCR cache configuration
gatein.jcr.cache.config=file:${gatein.conf.dir}/jcr/jbosscache/${gatein.jcr.config.type}/cache-config.xml
gatein.jcr.cache.expiration.time=15m

# JCR Locks configuration
gatein.jcr.lock.cache.config=file:${gatein.conf.dir}/jcr/jbosscache/${gatein.jcr.config.type}/lock-config.xml

# JCR Index configuration
gatein.jcr.index.cache.config=file:${gatein.conf.dir}/jcr/jbosscache/${gatein.jcr.config.type}/indexer-config.xml

# JGroups configuration
gatein.jgroups.jmxstatistics.enable=true
# for eXo Cache and IDM org-service (in cluster cache-config.xml files)
gatein.jgroups.config=${gatein.conf.dir}/jgroups/jgroups-udp.xml
# for JCR
gatein.jcr.jgroups.config=file:${gatein.jgroups.config}
{code}

By default, the nodes discovery is based on _UDP_, in which *JGroups* is responsible for the nodes identification through the UDP transport. The administrator can change the configuration of detection and ports in _jgroups-udp.xml_.

{docbook}
<para xml:id="ADM.Clustering.Advanced_configuration.Shared_file_system">
<citetitle>
<emphasis role="bold">
Shared file system
</emphasis>
</citetitle>
</para>
{docbook}

Optionally, if you need separate physical storage for JCR indexes and value storage files, it is possible to configure related paths, each to a separate shared file system:
{code}
gatein.jcr.storage.data.dir=/PATH/TO/SHARED/VALUES_FS/values
gatein.jcr.index.data.dir=/PATH/TO/SHARED/INDEX_FS/index
{code}

h2. Local JCR index in cluster {anchor:id=ADM.Clustering.Local_JCR_IndexRoleNotInToc}

{note}
JCR clustering with local index on each node is a new feature. Find more information about _Indexing in clustered environment_ in JCR reference documentation.
{note}

If the cluster is used with the local JCR index on each node, apply the following changes to the steps described above:
* configure index data to a local directory on each node:
{code}
gatein.jcr.index.data.dir=/PATH/TO/LOCAL/INDEX
{code}
* run the cluster with the additional profile named "{{cluster-index-local}}" by adding the following profile to the startup script in the _bin/gatein.sh_ file;
{code}
EXO_PROFILES="-Dexo.profiles=default,cluster,cluster-index-local"
{code}
Or, by using the following command with this additional profile.
{code}
./start_eXo.sh default,cluster,cluster-index-local
{code}

h1. FAQs of clustering {anchor:id=ADM.Clustering.Clustering_FAQ}

{docbook}
<citetitle>
<para>
<emphasis role="bold">
Q1. How to migrate from local to the cluster mode?
</emphasis>
</para>
</citetitle>
{docbook}
If you intend to migrate your production system from the local (non-cluster) to the cluster mode, follow these steps:

*1.* Update the configuration to the cluster mode as explained above on your main server.

*2.* Use the same configuration on other cluster nodes.

*3.* Move the index and value storage to the shared file system.

*4.* Start the cluster.


{docbook}
<citetitle>
<para>
<emphasis role="bold">
Q2. Why is startup failed with the "Port value out of range" error?
</emphasis>
</para>
</citetitle>
{docbook}
On Linux, your startup is failed if you encounter the following error:

{code}
[INFO] Caused by: java.lang.IllegalArgumentException: Port value out of range: 65536
{code}

This problem happens under specific circumstances when the JGroups networking library behind the clustering attempts to detect the IP to communicate with other nodes.

You need to verify:

* The host name is a valid IP address, served by one of the network devices, such as _eth0_, and _eth1_.

* The host name is NOT defined as _localhost_ or 127.0.0.1.


{docbook}
<citetitle>
<para>
<emphasis role="bold">
Q3. How to solve the "failed sending message to null" error?
</emphasis>
</para>
</citetitle>
{docbook}
If you encounter the following error when starting up in the cluster mode on Linux:

{code}
Dec 15, 2010 6:11:31 PM org.jgroups.protocols.TP down
SEVERE: failed sending message to null (44 bytes)
java.lang.Exception: dest=/228.10.10.10:45588 (47 bytes)
{code}

Be aware that clustering on Linux only works with IPv4. Therefore, when using a cluster under Linux, add the following property to the JVM parameters:
{code} -Djava.net.preferIPv4Stack=true {code}

{docbook}
<citetitle>
<para>
<emphasis role="bold">
Q3. How to hide JGroups protocol warnings in the log?
</emphasis>
</para>
</citetitle>
{docbook}
In the cluster mode, several eXo Platform subsystems, such as JCR, various caches, and organization service use shared JGroups transport;
and in case of being used by the default UDP transport, it might cause a side effect (a lot of warnings) like these below:

{code}
WARNING: discarded message from different group "gatein-idm-api-cluster" (our group is "gatein-idm-store-cluster"). Sender was 192.168.1.55:54232
Dec 16, 2011 4:46:09 PM org.jgroups.protocols.TP passMessageUp
WARNING: discarded message from different group "gatein-idm-store-cluster" (our group is "gatein-idm-api-cluster"). Sender was 192.168.1.55:63364
Dec 16, 2011 4:46:10 PM org.jgroups.protocols.TP passMessageUp
{code}

To hide such warnings, need to configure the Application Server logger in an appropriate way:
* Apache Tomcat: add the following lines to the _{{{${CATALINA_HOME}/conf/logging.properties}}}_ file:
{code:java}
org.jgroups.level = SEVERE
org.jgroups.handlers = java.util.logging.ConsoleHandler,6gatein.org.apache.juli.FileHandler
{code}
* JBoss Application Server: for the "_all_" server profile, add the following lines to the _{{{${jboss_server}/server/all/conf/jboss-log4j.xml}}}_ file:
{code:xml}
  <category name="org.jgroups.protocols.UDP">
    <priority value="ERROR"/>
  </category>
{code}

{docbook}
<para>
<prompt>
    <emphasis role="bold">See also</emphasis>
</prompt>
</para>
{docbook}
* [Set up the eXo Platform cluster|#ADM.Clustering.Cluster_setup]