Package org.exoplatform.documents.storage.jcr.search

Class DocumentTextExtractor

java.lang.Object
org.exoplatform.documents.storage.jcr.search.DocumentTextExtractor
public class DocumentTextExtractor extends Object
Extracts the text of a document file straight from its JCR binary, for when the search index does not hold it (yet): a file indexed a few seconds after it was saved, or never indexed at all. It reuses the platform's DocumentReaderService, the Tika-backed extractor the JCR full-text indexing already relies on, rather than Tika directly.

Bounded on every side, since it runs on a user's request: files of maxFileSizeBytes or larger are not read (the same 10 MB as the search index's own content extraction), only the file types the search index extracts the text of are read, the text is cut at maxChars, the caller stops waiting after timeoutMillis, and at most MAX_CONCURRENT_EXTRACTIONS run at a time with a short queue, anything beyond being refused at once rather than piling up.

What bounds memory is the file size times MAX_CONCURRENT_EXTRACTIONS: the readers configured for PDF and office files are not streaming ones and build the whole text before it is cut, and they cannot be interrupted, so an extraction the caller stopped waiting for keeps its slot until it ends.

The file is read with a system session: the caller checks the user can access the document before asking for its text.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    DEFAULT_SUPPORTED_MIME_TYPES
    The search index's own default for SUPPORTED_MIME_TYPES_PROPERTY, ecms core-search-configuration.xml.
    static final int
    MAX_CONCURRENT_EXTRACTIONS
    How many extractions run at the same time, at most.
    static final int
    MAX_QUEUED_EXTRACTIONS
    How many extractions may wait for a free slot before new ones are refused.
    static final String
    SUPPORTED_MIME_TYPES_PROPERTY
    The property listing, as regular expressions, the file types whose text the search index extracts.

  • Constructor Summary

    Constructors
    Constructor
    Description
    DocumentTextExtractor(org.exoplatform.services.jcr.RepositoryService repositoryService, org.exoplatform.services.document.DocumentReaderService documentReaderService, long maxFileSizeBytes, int maxChars, long timeoutMillis)
    Creates an extractor with its own bounded pool of extraction threads.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.exoplatform.documents.model.DocumentTextContent
    extract(String documentId)
    Extracts the text of a document file, within the time limit.
    protected javax.jcr.Session
    openSystemSession()
    Opens a system session on the workspace holding the documents.
    protected org.exoplatform.documents.model.DocumentTextContent
    readText(String documentId)
    Reads the text of a document file on the calling thread, with a system session opened and closed here since a session must not cross threads.
    void
    shutdown()
    Stops the extraction threads.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • MAX_CONCURRENT_EXTRACTIONS

      public static final int MAX_CONCURRENT_EXTRACTIONS
      How many extractions run at the same time, at most.
      See Also:

    • MAX_QUEUED_EXTRACTIONS

      public static final int MAX_QUEUED_EXTRACTIONS
      How many extractions may wait for a free slot before new ones are refused.
      See Also:

    • SUPPORTED_MIME_TYPES_PROPERTY

      public static final String SUPPORTED_MIME_TYPES_PROPERTY
      The property listing, as regular expressions, the file types whose text the search index extracts.
      See Also:

    • DEFAULT_SUPPORTED_MIME_TYPES

      public static final String DEFAULT_SUPPORTED_MIME_TYPES
      The search index's own default for SUPPORTED_MIME_TYPES_PROPERTY, ecms core-search-configuration.xml.
      See Also:

  • Constructor Details

    • DocumentTextExtractor

      public DocumentTextExtractor(org.exoplatform.services.jcr.RepositoryService repositoryService, org.exoplatform.services.document.DocumentReaderService documentReaderService, long maxFileSizeBytes, int maxChars, long timeoutMillis)
      Creates an extractor with its own bounded pool of extraction threads.
      Parameters:
      repositoryService - the JCR repository holding the files
      documentReaderService - the platform's text extractor
      maxFileSizeBytes - files of this size or larger are not read
      maxChars - the extracted text is cut at this length
      timeoutMillis - an extraction taking longer is abandoned

  • Method Details

    • extract

      public org.exoplatform.documents.model.DocumentTextContent extract(String documentId)
      Extracts the text of a document file, within the time limit.
      Parameters:
      documentId - the JCR identifier of the document
      Returns:
      the extracted text, or why there is none

    • readText

      protected org.exoplatform.documents.model.DocumentTextContent readText(String documentId) throws Exception
      Reads the text of a document file on the calling thread, with a system session opened and closed here since a session must not cross threads.
      Parameters:
      documentId - the JCR identifier of the document
      Returns:
      the extracted text, or why there is none
      Throws:
      Exception - when the repository cannot be reached

    • openSystemSession

      protected javax.jcr.Session openSystemSession() throws javax.jcr.RepositoryException
      Opens a system session on the workspace holding the documents.
      Returns:
      a new session, logged out by the caller
      Throws:
      javax.jcr.RepositoryException - when the repository cannot be reached

    • shutdown

      public void shutdown()
      Stops the extraction threads.