<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd">
<section id="JCR.PerformanceTuningGuide">
  <title>JCR Performance Tuning Guide</title>

  <para>This section will show you possible ways of improving JCR</para>

  <para>It is intended to GateIn Administrators and those who wants to use JCR
  features.</para>

  <section id="JCR.PerformanceTuningGuide.JCRPerformanceAndScalability">
    <title>JCR Performance and Scalability</title>

    <section id="JCR.PerformanceTuningGuide.JCRPerformanceAndScalability.ClusterConfiguration">
      <title>Cluster configuration</title>

      <para><citetitle>EC2 network</citetitle>: 1Gbit</para>

      <para><citetitle>Servers hardware</citetitle>:<simplelist>
          <member>7.5 GB memory</member>

          <member>4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute
          Units each)</member>

          <member>850 GB instance storage (2×420 GB plus 10 GB root
          partition)</member>

          <member>64-bit platform</member>

          <member>I/O Performance: High</member>

          <member>API name: m1.large</member>
        </simplelist></para>

      <note>
        <para>NFS and statistics (cacti snmp) server were located on one
        physical server.</para>
      </note>

      <para><citetitle>JBoss AS configuration</citetitle></para>

      <para><code>JAVA_OPTS: -Dprogram.name=run.sh -server -Xms4g -Xmx4g
      -XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true
      -Dsun.rmi.dgc.client.gcInterval=3600000
      -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseParallelGC
      -Djava.net.preferIPv4Stack=true</code></para>
    </section>

    <section id="JCR.PerformanceTuningGuide.JCRPerformanceAndScalability.JCRClusteredPerformance">
      <title>JCR Clustered Performance</title>

      <para>Benchmark test using webdav (Complex read/write load test
      (benchmark)) with 20K same file. To obtain per-operation results we have
      used custom output from the testscase threads to CSV file.</para>

      <para><citetitle>Read operation</citetitle>:<simplelist>
          <member>Warm-up iterations: 100</member>

          <member>Run iterations: 2000</member>

          <member>Background writing threads: 25</member>

          <member>Reading threads: 225</member>
        </simplelist></para>

      <mediaobject>
        <imageobject>
          <imagedata fileref="images/perf_EC2_results.jpg" />
        </imageobject>
      </mediaobject>

      <table>
        <title></title>

        <tgroup cols="4">
          <thead>
            <row>
              <entry>Nodes count</entry>

              <entry>tps</entry>

              <entry>Responses &gt;2s</entry>

              <entry>Responses &gt;4s</entry>
            </row>
          </thead>

          <tbody>
            <row>
              <entry>1</entry>

              <entry>523</entry>

              <entry>6.87%</entry>

              <entry>1.27%</entry>
            </row>

            <row>
              <entry>2</entry>

              <entry>1754</entry>

              <entry>0.64%</entry>

              <entry>0.08%</entry>
            </row>

            <row>
              <entry>3</entry>

              <entry>2388</entry>

              <entry>0.49%</entry>

              <entry>0.09%</entry>
            </row>

            <row>
              <entry>4</entry>

              <entry>2706</entry>

              <entry>0.46%</entry>

              <entry>0.1%</entry>
            </row>
          </tbody>
        </tgroup>
      </table>

      <para><citetitle>Read operaion with more threads</citetitle>:</para>

      <simplelist>
        <member>Warm-up iterations: 100</member>

        <member>Run iterations: 2000</member>

        <member>Background writing threads: 50</member>

        <member>Reading threads: 450</member>
      </simplelist>

      <mediaobject>
        <imageobject>
          <imagedata fileref="images/perf_EC2_results_2.jpg" />
        </imageobject>
      </mediaobject>

      <table>
        <title></title>

        <tgroup cols="4">
          <thead>
            <row>
              <entry>Nodes count</entry>

              <entry>tps</entry>

              <entry>Responses &gt;2s</entry>

              <entry>Responses &gt;4s</entry>
            </row>
          </thead>

          <tbody>
            <row>
              <entry>1</entry>

              <entry>116</entry>

              <entry>?</entry>

              <entry>?</entry>
            </row>

            <row>
              <entry>2</entry>

              <entry>1558</entry>

              <entry>6.1%</entry>

              <entry>0.6%</entry>
            </row>

            <row>
              <entry>3</entry>

              <entry>2242</entry>

              <entry>3.1%</entry>

              <entry>0.38%</entry>
            </row>

            <row>
              <entry>4</entry>

              <entry>2756</entry>

              <entry>2.2%</entry>

              <entry>0.41%</entry>
            </row>
          </tbody>
        </tgroup>
      </table>
    </section>
  </section>

  <section id="JCR.PerformanceTuningGuide.PerformanceTuningGuide">
    <title>Performance Tuning Guide</title>

    <section id="JCR.PerformanceTuningGuide.PerformanceTuningGuide.JBossASTuning">
      <title>JBoss AS Tuning</title>

      <para>You can use <parameter>maxThreads</parameter> parameter to
      increase maximum amount of threads that can be launched in AS instance.
      This can improve performance if you need a high level of concurrency.
      also you can use <code>-XX:+UseParallelGC</code> java directory to use
      parallel garbage collector.</para>

      <tip>
        <para>Beware of setting <parameter>maxThreads</parameter> too big,
        this can cause <exceptionname>OutOfMemoryError</exceptionname>. We've
        got it with <code>maxThreads=1250</code> on such machine:</para>

        <simplelist>
          <member>7.5 GB memory</member>

          <member>4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute
          Units each)</member>

          <member>850 GB instance storage (2×420 GB plus 10 GB root
          partition)</member>

          <member>64-bit platform</member>

          <member>I/O Performance: High</member>

          <member>API name: m1.large</member>

          <member>java -Xmx 4g</member>
        </simplelist>
      </tip>
    </section>

    <section id="JCR.PerformanceTuningGuide.PerformanceTuningGuide.Clustering">
      <title>Clustering</title>

      <para>For performance it is better to have loadbalacer, DB server and
      shared NFS on different computers. If for some reasons you see that one
      node gets more load than others you can decrease this load using load
      value in the configuration of your load balancer.</para>

      <para><citetitle>JGroups configuration</citetitle></para>

      <para>It's recommended to use the JGroups shared transaport. It is
      configured by default in eXo JCR and offers higher performance in
      cluster, using less network connections also. If there are two or more
      clusters in your network, please check that they use different ports and
      different cluster names.</para>

      <para><citetitle>Write performance in cluster</citetitle></para>

      <para>Exo JCR implementation uses Lucene indexing engine to provide
      search capabilities. But Lucene brings some limitations for write
      operations: it can perform indexing only in one thread. Thats why write
      performance in cluster is not higher than in singleton environment. Data
      is indexed on coordinator node, so increasing write-load on cluster may
      lead to ReplicationTimeout exception. It occurs because writing threads
      queue in the indexer and under high load timeout for replication to
      coordinator will be exceeded.</para>

      <para>Taking in consideration this fact, it is recommended to exceed
      <parameter>replTimeout</parameter> value in cache configurations in case
      of high write-load.</para>

      <para><citetitle>Replication timeout</citetitle></para>

      <para>Some operations may take too much time. So if you get
      <exceptionname>TimeoutException</exceptionname> try to increase the
      replication timeout:</para>

      <programlisting language="xml">      &lt;clustering mode="replication"&gt;
        ...
        &lt;sync replTimeout="20000"/&gt;
      &lt;/clustering&gt;
   </programlisting>

      <para>value is set in miliseconds.</para>

      <para>Another thing that you should check if you get
      <exceptionname>TimeoutException</exceptionname> is the jgroups thread
      pools (for normal messages and out-of-band messages). Indeed if one of
      them is exhausted you can easily get this kind of exceptions. To know if
      they are exhausted, you can get a thread dump using the jstack command
      on your process id and check if you have some unused jgroups threads
      knowing that the name of the threads by default starts with "Incoming-"
      followed by the thread index in case of the thread pool for normal
      messages and starts with "OOB-" followed by the thread index in case of
      the thread pool for out-of-band messages. See below an example of an
      unused thread for normal messages and for out-of-band messages:</para>

      <programlisting>"OOB-1,shared=JCR-cluster" prio=5 tid=7fbfd00a5000 nid=0x117bb0000 waiting on condition [117baf000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  &lt;77aae61e8&gt; (a java.util.concurrent.SynchronousQueue$TransferStack)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
 at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
 at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
 at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
 at java.lang.Thread.run(Thread.java:680)


"Incoming-1,shared=JCR-cluster" prio=5 tid=7fbfce5d3800 nid=0x119ecd000 waiting on condition [119ecc000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  &lt;77abd6568&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
 at java.lang.Thread.run(Thread.java:680)</programlisting>

      <para>In case you realize that at least one of the thread pools is
      exhausted, you will need to increase the size of the corresponding
      thread pool in your jgroups configuration. You need also to make sure
      that you allocated enough RAM to be able to support the total amount of
      added threads. For each thread pool, you can define the min and max
      amount of threads but also the keep alive time, the queue max size and
      the rejection policy. The configuration of thread pools is at protocol
      configuration level, in the next example, we use TCP so it is at TCP
      configuration level: </para>

      <programlisting language="xml">    &lt;UDP
         singleton_name="JCR-cluster" 
         ...

         thread_pool.enabled="true"
         thread_pool.min_threads="6"
         thread_pool.max_threads="24"
         thread_pool.keep_alive_time="5000"
         thread_pool.queue_enabled="true"
         thread_pool.queue_max_size="10000"
         thread_pool.rejection_policy="discard"

         oob_thread_pool.enabled="true"
         oob_thread_pool.min_threads="3"
         oob_thread_pool.max_threads="24"
         oob_thread_pool.keep_alive_time="5000"
         oob_thread_pool.queue_enabled="false"
         oob_thread_pool.queue_max_size="100"
         oob_thread_pool.rejection_policy="Run"/&gt;</programlisting>
    </section>
  </section>
</section>
