eXo Platform 3.5

## Chapter 50. JCR Performance Tuning Guide

## Introduction

This guide will show you possible ways of improving JCR performance.

It is intended to GateIn Administrators and those who wants to use JCR features.

## JCR Performance and Scalability

### Cluster configuration

EC2 network: 1Gbit

Servers hardware:

7.5 GB memory
4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
850 GB instance storage (2×420 GB plus 10 GB root partition)
64-bit platform
I/O Performance: High
API name: m1.large

### Note

NFS and statistics (cacti snmp) server were located on one physical server.

JBoss AS configuration

`JAVA_OPTS: -Dprogram.name=run.sh -server -Xms4g -Xmx4g -XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseParallelGC -Djava.net.preferIPv4Stack=true`

### JCR Clustered Performance

Benchmark test using webdav (Complex read/write load test (benchmark)) with 20K same file. To obtain per-operation results we have used custom output from the testscase threads to CSV file.

Read operation:

Warm-up iterations: 100
Run iterations: 2000
Background writing threads: 25
Reading threads: 225

Nodes count | tps | Responses >2s | Responses >4s
--- | --- | --- | ---
1 | 523 | 6.87% | 1.27%
2 | 1754 | 0.64% | 0.08%
3 | 2388 | 0.49% | 0.09%
4 | 2706 | 0.46% | 0.1%

Read operaion with more threads:

Warm-up iterations: 100
Run iterations: 2000
Background writing threads: 50
Reading threads: 450

Nodes count | tps | Responses >2s | Responses >4s
--- | --- | --- | ---
1 | 116 | ? | ?
2 | 1558 | 6.1% | 0.6%
3 | 2242 | 3.1% | 0.38%
4 | 2756 | 2.2% | 0.41%

## Performance Tuning Guide

### JBoss AS Tuning

You can use `maxThreads` parameter to increase maximum amount of threads that can be launched in AS instance. This can improve performance if you need a high level of concurrency. also you can use `-XX:+UseParallelGC` java directory to use paralel garbage collector.

### Tip

Beware of setting `maxThreads` too big, this can cause `OutOfMemoryError`. We've got it with `maxThreads=1250` on such machine:

7.5 GB memory
4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
850 GB instance storage (2×420 GB plus 10 GB root partition)
64-bit platform
I/O Performance: High
API name: m1.large
java -Xmx 4g

### JCR Cache Tuning

Cache size

JCR-cluster implementation is built using JBoss Cache as distributed, replicated cache. But there is one particularity related to remove action in it. Speed of this operation depends on the actual size of cache. As many nodes are currently in cache as much time is needed to remove one particular node (subtree) from it.

Eviction

Manipulations with eviction `wakeUpInterval` value doestn't affect on performance. Performance results with values from 500 up to 3000 are approximately equal.

Transaction Timeout

Using short timeout for long transactions such as Export/Import, removing huge subtree defined timeout may cause `TransactionTimeoutException`. [TODO] put recomended timeout value

### Clustering

For performance it is better to have loadbalacer, DB server and shared NFS on different computers. If in some reasons you see that one node gets more load than others you can decrease this load using load value in load balancer.

JGroups configuration

It's recommended to use "multiplexer stack" feature present in JGroups. It is set by default in eXo JCR and offers higher performance in cluster, using less network connections also. If there are two or more clusters in your network, please check that they use different ports and different cluster names.

Write performance in cluster

Exo JCR implementation uses Lucene indexing engine to provide search capabilities. But Lucene brings some limitations for write operations: it can perform indexing only in one thread. Thats why write performance in cluster is not higher than in singleton environment. Data is indexed on coordinator node, so increasing write-load on cluster may lead to ReplicationTimeout exception. It occurs because writing threads queue in the indexer and under high load timeout for replication to coordinator will be exceeded.

Taking in consideration this fact, it is recommended to exceed `replTimeout` value in cache configurations in case of high write-load.

Replication timeout

Some operations may take too much time. So if you get `ReplicationTimeoutException` try increasing replication timeout:

       <clustering mode="replication" clusterName="${jbosscache-cluster-name}">

          ...

          <sync replTimeout="60000" />

       </clustering>

value is set in miliseconds.

### JVM parameters

PermGen space size

If you intend to use Infinispan, you will have to increase the PermGen size to at least 256 Mo due to the latest versions of JGroups that are needed by Infinispan (please note that Infinspan is only dedicated to the community for now, no support will be provided). In case, you intend to use JBoss Cache, you can keep on using JGroups 2.6.13.GA which means that you don't need to increase the PermGen size.